Providing at least one peer connection between a plurality of coupling facilities to couple the plurality of coupling facilities

ABSTRACT

A coupling facility is coupled to one or more other coupling facilities via one or more peer links. The coupling of the facilities enables various functions to be supported, including the duplexing of structures of the coupling facilities. Duplexing is performed on a structure basis, and thus, a coupling facility may include duplexed structures, as well as non-duplexed or simplexed structures.

CROSS-REFERENCE TO RELATED APPLICATIONS/PATENTS

This application contains subject matter which is related to the subjectmatter of the following applications/patents, each of which is assignedto the same assignee as this application. Each of the below listedapplications/patents is hereby incorporated herein by reference in itsentirety:

“TEST TOOL AND METHOD FOR FACILITATING TESTING OF DUPLEXED COMPUTERFUNCTIONS”, Jones et al., Ser. No. 09/968,420, filed Oct. 1, 2001;

“RESTARTING A COUPLING FACILITY COMMAND USING A TOKEN FROM ANOTHERCOUPLING FACILITY COMMAND”, Elko et al., Ser. No. 09/968,729, filed Oct.1, 2001;

“DYNAMICALLY DETERMINING WHETHER TO PROCESS REQUESTS SYNCHRONOUSLY ORASYNCHRONOUSLY”, Jordan et al., Ser. No. 09/968,185, filed Oct. 1, 2001;

“MANAGING THE STATE OF COUPLING FACILITY STRUCTURES”, Elko et al., Ser.No. 09/968,248, filed Oct. 1, 2001;

“SYNCHRONIZING PROCESSING OF COMMANDS INVOKED AGAINST DUPLEXED COUPLINGFACILITY STRUCTURES”, Elko et al., Ser. No. 09/968,179, filed Oct. 1,2001;

“SYSTEM-MANAGED DUPLEXING OF COUPLING FACILITY STRUCTURES”, Allen etal., Ser. No. 09/968,242, filed Oct. 1, 2001;

“METHOD, SYSTEM AND PROGRAM PRODUCTS FOR PROVIDING USER-MANAGEDDUPLEXING OF COUPLING FACILITY CACHE STRUCTURES”, Elko et al., Ser. No.09/255,382, filed Feb. 22, 1999;

“CASTOUT PROCESSING FOR DUPLEXED CACHE STRUCTURES”, Elko et al., Ser.No. 09/255,383, filed Feb. 22, 1999;

“SYSTEM-MANAGED REBUILD OF COUPLING FACILITY STRUCTURES”, Allen et al.,Ser. No. 09/378,780, filed Aug. 23, 1999;

“METHOD, SYSTEM AND PROGRAM PRODUCTS FOR COPYING COUPLING FACILITYSTRUCTURES”, Allen et al., Ser. No. 09/379,054, filed Aug. 23, 1999;

“METHOD, SYSTEM AND PROGRAM PRODUCTS FOR MODIFYING COUPLING FACILITYSTRUCTURES”, Dahlen et al., Ser. No. 09/379,435, filed Aug. 23, 1999;

“DIRECTED ALLOCATION OF COUPLING FACILITY STRUCTURES”, Dahlen et al.,Ser. No. 09/378,861, filed Aug. 23, 1999;

“METHOD, SYSTEM AND PROGRAM PRODUCTS FOR COPYING COUPLING FACILITY LOCKSTRUCTURES”, Allen et al., Ser. No. 09/379,053, filed Aug. 23, 1999;

“METHOD OF CONTROLLING THE FLOW OF INFORMATION BETWEEN SENDERS ANDRECEIVERS ACROSS LINKS BEING USED AS CHANNELS”, Gregg et al., Ser. No.09/151,051, filed Sep. 10, 1998;

“SYSTEM OF CONTROLLING THE FLOW OF INFORMATION BETWEEN SENDERS ANDRECEIVERS ACROSS LINKS BEING USED AS CHANNELS”, Gregg et al., Ser. No.09/150,942, filed Sep. 10, 1998;

“SYSTEM OF PERFORMING PARALLEL CLEANUP OF SEGMENTS OF A LOCK STRUCTURELOCATED WITHIN A COUPLING FACILITY”, Dahlen et al., U.S. Pat. No.6,233,644 B1, issued May 15, 2001;

“MULTI CHANNEL INTER-PROCESSOR COUPLING FACILITY PROCESSING RECEIVEDCOMMANDS STORED IN MEMORY ABSENT STATUS ERROR OF CHANNELS”, Elko et al.,U.S. Pat. No. 5,574,945, issued Nov. 12, 1996;

“METHOD, SYSTEM AND PROGRAM PRODUCTS FOR MANAGING CHANGED DATA OFCASTOUT CLASSES”, Elko et al., U.S. Pat. No. 6,230,243 B1, issued May 8,2001;

“METHOD AND SYSTEM FOR CAPTURING AND CONTROLLING ACCESS TO INFORMATIONIN A COUPLING FACILITY”, Neuhard et al., U.S. Pat. No. 5,630,050, issuedMay 13, 1997;

“DYNAMICALLY ASSIGNING A DUMP SPACE IN A SHARED DATA FACILITY TO RECEIVEDUMPING INFORMATION TO BE CAPTURED”, Elko et al., U.S. Pat. No.5,664,155, issued Sep. 2, 1997;

“METHOD AND APPARATUS FOR DISTRIBUTED LOCKING OF SHARED DATA, EMPLOYINGA CENTRAL COUPLING FACILITY”, Elko et al., U.S. Pat. No. 5,339,427,issued Aug. 16, 1994;

“METHOD AND SYSTEM FOR LOG MANAGEMENT IN A COUPLED DATA PROCESSINGSYSTEM”, Geiner et al., U.S. Pat. No. 5,737,600, issued Apr. 7, 1998;

“METHOD OF PERFORMING PARALLEL CLEANUP OF SEGMENTS OF A LOCK STRUCTURE”,Dahlen et al., U.S. Pat. No. 6,178,421 B1, issued Jan. 23, 2001;

“SPEEDING-UP COMMUNICATION RATES ON LINKS TRANSFERRING DATA STRUCTURESBY A METHOD OF HANDING SCATTER/GATHER OF STORAGE BLOCKS IN COMMANDEDCOMPUTER SYSTEMS”, Gregg et al., U.S. Pat. No. 5,948,060, issued Sep. 7,1999;

“METHOD OF MANAGING RESOURCES IN ONE OR MORE COUPLING FACILITIES COUPLEDTO ONE OR MORE OPERATING SYSTEMS IN ONE OR MORE CENTRAL PROGRAMMINGCOMPLEXES USING A POLICY”, Allen et al., U.S. Pat. No. 5,634,072, issuedMay 27, 1997;

“METHOD AND APPARATUS FOR OPTIMIZING THE HANDLING OF SYNCHRONOUSREQUESTS TO A COUPLING FACILITY IN A SYSPLEX CONFIGURATION”, Kubala etal., U.S. Pat. No. 5,923,890, issued Jul. 13, 1999;

“METHOD FOR RECEIVING MESSAGES AT A COUPLING FACILITY”, Elko et al.,U.S. Pat. No. 5,706,432, issued Jan. 6, 1998;

“COMMAND EXECUTION SYSTEM FOR USING FIRST AND SECOND COMMANDS TO RESERVEAND STORE SECOND COMMAND RELATED STATUS INFORMATION IN MEMORY PORTIONRESPECTIVELY”, Elko et al., U.S. Pat. No. 5,392,397, issued Feb. 21,1995;

“SOFTWARE CACHE MANAGEMENT OF A SHARED ELECTRONIC STORE IN A SUPPLEX”,Elko et al., U.S. Pat. No. 5,457,793, issued Oct. 10, 1995;

“REQUESTING A DUMP OF INFORMATION STORED WITHIN A COUPLING FACILITY, INWHICH THE DUMP INCLUDES SERVICEABILITY INFORMATION FROM AN OPERATINGSYSTEM THAT LOST COMMUNICATION WITH THE COUPLING FACILITY”, Neuhard etal, U.S. Pat. No. 5,860,115, issued Jan. 12, 1999;

“AUTHORIZATION METHOD FOR CONDITIONAL COMMAND EXECUTION”, Elko et al,U.S. Pat. No. 5,450,590, issued Sep. 12, 1995;

“IN A MULTIPROCESSING SYSTEM HAVING A COUPLING FACILITY, COMMUNICATINGMESSAGES BETWEEN THE PROCESSORS AND THE COUPLING FACILITY IN EITHER ASYNCHRONOUS OPERATION OR AN ASYNCHRONOUS OPERATION”, Elko et al., U.S.Pat. No. 5,561,809, issued Oct. 1, 1996;

“COUPLING FACILITY FOR RECEIVING COMMANDS FROM PLURALITY OF HOSTS FORACTIVATING SELECTED CONNECTION PATHS TO I/O DEVICES AND MAINTAININGSTATUS THEREOF”, Elko et al., U.S. Pat. No. 5,463,736, issued Oct. 31,1995;

“METHOD AND SYSTEM FOR MANAGING DATA AND USERS OF DATA IN A DATAPROCESSING SYSTEM”, Allen et al., U.S. Pat. No. 5,465,359, issued Nov.7, 1995;

“METHODS AND SYSTEMS FOR CREATING A STORAGE DUMP WITHIN A COUPLINGFACILITY OF A MULTISYSTEM ENVIRONMENT”, Elko et al., U.S. Pat. No.5,761,739, issued Jun. 2, 1998;

“METHOD AND APPARATUS FOR COUPLING DATA PROCESSING SYSTEMS”, Elko etal., U.S. Pat. No. 5,317,739, issued May 31, 1994;

“METHOD AND APPARATUS FOR EXPANSION, CONTRACTION, AND REAPPORTIONMENT OFSTRUCTURED EXTERNAL STORAGE STRUCTURES”, Dahlen et al., U.S. Pat. No.5,581,737, issued Dec. 3, 1996;

“SYSPLEX SHARED DATA COHERENCY METHOD”, Elko et al., U.S. Pat. No.5,537,574, issued Jul. 16, 1996;

“MULTIPLE PROCESSOR SYSTEM HAVING SOFTWARE FOR SELECTING SHARED CACHEENTRIES ON AN ASSOCIATED CASTOUT CLASS FOR TRANSFER TO A DASD WITH ONEI/O OPERATION”, Elko et al., U.S. Pat. No. 5,493,668, issued Feb. 20,1996;

“INTEGRITY OF DATA OBJECTS USED TO MAINTAIN STATE INFORMATION FOR SHAREDDATA AT A LOCAL COMPLEX”, Elko et al., U.S. Pat. No. 5,331,673, issuedJul. 19, 1994;

“COMMAND QUIESCE FUNCTION”, Elko et al., U.S. Pat. No. 5,339,405, issuedAug. 16, 1994;

“METHOD AND APPARATUS FOR PERFORMING CONDITIONAL OPERATIONS ONEXTERNALLY SHARED DATA”, Elko et al., U.S. Pat. No. 5,742,830, issuedApr. 21, 1998;

“METHOD AND SYSTEM FOR RECONFIGURING A STORAGE STRUCTURE WITHIN ASTRUCTURE PROCESSING FACILITY”, Allen et al., U.S. Pat. No. 5,515,499,issued May 7, 1996;

“METHOD FOR COORDINATING EXECUTING PROGRAMS IN A DATA PROCESSINGSYSTEM”, Allen et al., U.S. Pat. No. 5,604,863, issued Feb. 18, 1997;and

“SYSTEM AND METHOD FOR MANAGEMENT OF OBJECT TRANSITIONS IN AN EXTERNALSTORAGE FACILITY ACCESSED BY ONE OR MORE PROCESSORS”, Dahlen et al.,U.S. Pat. No. 5,887,135, issued Mar. 23, 1999.

TECHNICAL FIELD

This invention relates, in general, to data processing within adistributed computing environment, and in particular, to the coupling ofa plurality of coupling facilities of the distributed computingenvironment for various purposes, including the duplexing of structureswithin the coupling facilities.

BACKGROUND OF THE INVENTION

Some distributed computing environments, such as Parallel Sysplexes,today provide a non-volatile shared storage device called the couplingfacility, that includes multiple storage structures of either the cacheor list type. These structures provide unique functions for theoperating system and middleware products employed for the efficientoperation of a Parallel Sysplex. For example, the cache structuresprovide directory structures and cross-invalidation mechanisms tomaintain buffer coherency for multisystem databases, as well as a fastwrite medium for database updates. These are used by, for instance, thedata sharing versions of DB2 and IMS, offered by International BusinessMachines Corporation, Armonk, N.Y.

The list structures provide many diverse functions. One such liststructure function is to provide for high-performance global locking,and this function is exploited by such products as the IMS Resource LockManager (IRLM) and the Global Resource Serialization (GRS) function inOS/390, offered by International Business Machines Corporation, Armonk,N.Y. Another list structure function is to provide a message passingmechanism with storage for maintaining multiple messages on a per systembasis and a mechanism for notifying a system of the arrival of newmessages. This function is exploited by the XCF component of OS/390,which in turn is exploited by numerous multisystem applications forproviding a capability to pass messages between their various instances.A third list structure function is to provide for shared queuestructures that can be ordered and accessed by LIFO/FIFO ordering, bykey, or by name. Workload Manager (WLM), IMS Shared Message Queues andMQ Series, all offered by International Business Machines Corporation,Armonk, N.Y., are examples of exploiters of this feature. While thesefunctions provide examples of the list structure uses, other uses exist.

Various components of a Parallel Sysplex have been documented innumerous applications/patents, which are listed above and herebyincorporated herein by reference in their entirety. The capabilitiesdefined in some of those patents provide the basic system structure tocreate and manage cache and list structure instances. Additionally,various of the applications/patents listed above provide extensions tothe base functions of the Parallel Sysplex.

In many situations, a failure of the coupling facility that containsvarious structures requires significant recovery actions to be taken bythe owning applications. For example, for database caches and queues,this may require using backup log data sets and/or tapes. This is atime-consuming process that results in a loss of access to theapplication during the recovery operation. Other structures, such aslock tables, may require reconstruction of partial lock tables fromin-storage copies, along with failures of in-flight transactions. Stillother structures, such as message-passing structures, may lose all dataand require re-entry from the application. So, there is a proliferationof diverse recovery schemes with different recovery times and impacts.Moreover, since the failure of a coupling facility results in allresident structures failing, the diverse recovery actions are occurringconcurrently, which can cause serious disruptions in the ParallelSysplex.

Thus, a need exists for a configuration of a Parallel Sysplex thatprovides less disruptions. In particular, a need exists for ahigh-availability coupling facility, which improves on the recoverytimes and impacts of existing recovery techniques, while also providesfor a consistent recovery design across various structure types. As aparticular example, a need exists for a capability that enables couplingfacilities to be coupled to one another to provide various capabilities,including, but not limited to, the duplexing of structures in separatecoupling facilities.

SUMMARY OF THE INVENTION

The shortcomings of the prior art are overcome and additional advantagesare provided through the provision of a method of coupling a pluralityof coupling facilities of a computing environment. The method includes,for instance, selecting a first coupling facility and a second couplingfacility to be coupled; and providing at least one peer connectionbetween the first coupling facility and the second coupling facility tocouple the first coupling facility and the second coupling facility.

In a further embodiment, a method of coupling a plurality of couplingfacilities of a computing environment is provided. The method includes,for instance, selecting a plurality of coupling facilities to becoupled; and providing at least one peer connection between theplurality of coupling facilities.

System and computer program products corresponding to theabove-summarized methods are also described and claimed herein.

Additional features and advantages are realized through the techniquesof the present invention. Other embodiments and aspects of the inventionare described in detail herein and are considered a part of the claimedinvention.

BRIEF DESCRIPTION OF THE DRAWINGS

The subject matter which is regarded as the invention is particularlypointed out and distinctly claimed in the claims at the conclusion ofthe specification. The foregoing and other objects, features, andadvantages of the invention are apparent from the following detaileddescription taken in conjunction with the accompanying drawings inwhich:

FIG. 1 depicts one example of a duplexing model in which a plurality ofcoupling facilities are coupled to one another via a peer link, inaccordance with an aspect of the present invention;

FIG. 2 depicts one alternative embodiment of a duplexing model, whichuses an explicit lock for duplexing;

FIG. 3 depicts another alternative embodiment of a duplexing model withstore-and-forward duplexing;

FIG. 4 depicts one embodiment of a Sysplex configuration that couples aplurality of coupling facilities, in accordance with an aspect of thepresent invention;

FIG. 5 depicts one embodiment of a message-path status vector, inaccordance with an aspect of the present invention;

FIG. 6 depicts one embodiment of a signaling vector, in accordance withan aspect of the present invention;

FIG. 7 depicts one embodiment of a message-path configuration, inaccordance with an aspect of the present invention;

FIG. 8 depicts one example of a retry buffer format, in accordance withan aspect of the present invention;

FIG. 9 depicts one embodiment of the logic associated with activatingduplexing, in accordance with an aspect of the present invention;

FIG. 10 depicts one embodiment of the logic associated with deactivatingduplexing, in accordance with an aspect of the present invention;

FIG. 11 depicts one embodiment of the logic associated with a ProbeRemote Facility Connection command, in accordance with an aspect of thepresent invention;

FIG. 12 depicts one embodiment of the logic associated with a Set RetryBuffer Authority command, in accordance with an aspect of the presentinvention;

FIGS. 13 a and 13 b depict one embodiment of the logic associated withthe control flow for an Interface Control Check (IFCC), in accordancewith an aspect of the present invention;

FIG. 14 depicts one example of a duplexing signal format, in accordancewith an aspect of the present invention;

FIG. 15 depicts one embodiment of a mapping of retry buffers and asignaling vector, in accordance with an aspect of the present invention;

FIGS. 16 a and 16 b depict one embodiment of an overview of a couplingfacility duplexing signal protocol, in accordance with an aspect of thepresent invention;

FIG. 17 depicts one embodiment of the logic associated with a duplexcommand decode checking function, in accordance with an aspect of thepresent invention;

FIG. 18 depicts one embodiment of the logic associated with a Ready toExecute (RTE) exchange function, in accordance with an aspect of thepresent invention;

FIG. 19 depicts one embodiment of the logic associated with a simplexlatch resource function, in accordance with an aspect of the presentinvention;

FIGS. 20 a and 20 b depict one embodiment of the logic associated with aduplex latch resource function, in accordance with an aspect of thepresent invention; and

FIG. 21 depicts one embodiment of the logic associated with a Ready toComplete (RTC) exchange function, in accordance with an aspect of thepresent invention.

BEST MODE FOR CARRYING OUT THE INVENTION

In accordance with an aspect of the present invention, at least onecoupling facility is coupled to one or more other coupling facilitiesusing one or more peer connections. The coupling of the couplingfacilities enables a variety of capabilities to be performed, including,but not limited to, the duplexing of coupling facility structures.

The duplexing of the structures is performed in a manner that istransparent to the users of the structures. That is, when a user of astructure issues a command against the structure, the user is unawarethat the structure and thus, the command, are duplexed.

In one aspect of the present invention, a high-availability design ofcoupling facility structures is provided by duplexing a desiredstructure in two separate coupling facilities. This design improves onthe recovery times and impacts of existing recovery techniques, whilealso provides for a consistent recovery design across the variousstructure types.

In one embodiment, various aspects of duplexing are accomplished byextending the architecture and physical configuration of components ofthe Parallel Sysplex. For instance, a coupling facility-to-couplingfacility peer connection is provided that allows for protocol flowsbetween the coupling facilities through a signaling mechanism. As afurther example, a mechanism is provided for the operating systemcomponent of z/OS, the Locking Facility Support Services (LFSS)component, to create a duplexed copy of a given structure in twoseparate coupling facilities, and to then split commands into twoseparate commands, each sent in parallel to the two structure instances.This provides for parallel execution of the commands and for efficientre-execution of the commands on congested links. Moreover, in a furtheraspect of the present invention, a protocol extension to the commandconcurrency mechanism in the coupling facility is provided, whichexchanges signals on the peer connection in a way that maintainssynchronization of the structure contents for each command that isexecuted. These functions are provided in a manner, which has optimalSysplex performance, in one example.

The duplexing state of a cache or list structure is maintained in aglobal control, referred to as the duplexing vector, which is indexed bya Structure Identifier (SID). When the bit corresponding to a SID valueis B′1′, duplexing is active and the duplexing controls in the structureare valid. The duplexing controls include a duplexing system identifierand a duplexing system signaling vector token, as described below. Thesecontrols are used in conjunction with information provided in a messagecommand block (MCB) of a duplexed command to construct a duplexingsignal.

In one embodiment, structures are not differentiated as primary orsecondary within the structure. Primary or secondary structures aredetermined by how they are used. In some cases, new request operands aredefined to allow certain actions in the secondary to be suppressed, butthese are done on a command basis and do not create new states in thestructure. This simplifies recovery when a primary structure fails,since no state changes are needed in the secondary to make it act as aprimary.

While a secondary structure may be promoted to a primary structure, theopposite is not true, in this example. That is, a primary structure isnot demoted to a secondary structure.

Duplexing Models

Three architectural models of structure duplexing are possible that meetthe attributes of application transparency, failure isolation andcommand concurrency across two structures. These models include, forinstance:

(1) A model that includes a coupling facility-to-coupling facilitysignaling protocol that coordinates command completion across thecoupling facilities during command execution (FIG. 1).

(2) A model that includes an external locking protocol that serializescommand execution explicitly (FIG. 2). (A variation of this protocolthat satisfies the desired attributes of failure isolation and commandconcurrency for a certain type of structure, but is not applicationtransparent is described in co-pending U.S. patent application Ser. No.09/255,382 entitled “Method, System and Program for Providing aUser-Managed Duplexing of Coupling Facility Cache Structures”, filedFeb. 22, 1999, which is hereby incorporated herein by reference in itsentirety.)

(3) A store-and-forward model where the primary coupling facilityforwards each command on a standard ISC channel connected to a remotecoupling facility. The remote coupling facility then executes thecommand and returns the result to the primary coupling facility, whichthen completes the command to the operating system. This model isdepicted in FIG. 3.

Each of the models is described below.

(1) Coupling Facility-to-Coupling Facility Duplexing Model:

FIG. 1 depicts one embodiment of a configuration 100, which includes twocoupling facilities 102, 104 coupled to a system 106 in a ParallelSysplex. In one example, the system is running an instance of the z/OSoperating system 108, offered by International Business MachinesCorporation, Armonk, N.Y. Further, in one example, the system is runningan application 110 that is coupled to a coupling facility structure 112(either of a cache or list type), whose location is not known to theapplication. The actual physical connection is managed by a LockingFacility Support Services (LFSS) component 114 of the z/OS operatingsystem and commands initiated by the user application flow through theLFSS component.

Two instances of the coupling facility structure are maintained inseparate coupling facilities, referred to as the primary couplingfacility and the secondary coupling facility. A peer connection 116,such as an Intersystem Channel (ISC) link, couples the two couplingfacilities. The peer ISC link can transmit both primary message commandsand secondary message commands in either direction. This may bephysically represented by either two unidirectional links, one with asender channel on the primary coupling facility and a receiver channelon the secondary coupling facility, and the second link oppositelyconfigured. This may also be represented by a single physical link wherethe channel interface in each coupling facility supports both sender andreceiver functionality. This latter capability exists in ISC3 links andtheir variants: ICB3 and IC3, all of which are offered by InternationalBusiness Machines Corporation, Armonk, N.Y.

The peer ISC link between the coupling facilities is used, for instance,to exchange message path commands on the primary message commandinterface to configure and couple the two coupling facilities. Onceconfigured and coupled, the peer ISC link is also used to send secondarycommands of the list-notification type to exchange signals as part of asignaling protocol for command execution. The sending and receiving ofthese secondary commands is managed by a coupling facility componentcalled a signaling protocol engine 118. Requests by the cache and listcomponent of the coupling facility for sending and receiving duplexingsignals flow through the signaling protocol engine.

One embodiment of the steps in a normal command execution for thecoupling facility-to-coupling facility duplexing model are shown in FIG.1 in numeric sequence that approximates the time sequence of thecommand. In these steps, various components of the signaling protocolare described. A more complete description of the signaling protocol isprovided in a following section. The extensions to the protocoldescribed later are used to enhance the performance and reliability ofthe basic protocol described here.

Step 1. The user application generates a command and communicates thiscommand to the LFSS through a system macro interface.

Step 2. The LFSS creates two copies of the command, sending one to theprimary coupling facility and the second to the secondary couplingfacility. The LFSS uses an asynchronous SEND MESSAGE interface withoutnotification to allow the two commands to be initiated in parallel. TheLFSS also sets a synchronous completion on initial status (SCIS) bit ofthe SEND MESSAGE to minimize the effects of any busy conditionsencountered on the channel interface. A link-subsystem (LSS) component120 of the coupling facility control code (CFCC) in the primary couplingfacility receives the command and transfers control to the cache or listcomponent, as appropriate. Likewise, the link-subsystem (LSS) componentin the secondary coupling facility receives the command and transferscontrol to the cache or list component, as appropriate.

Step 3. The cache/list component of the primary coupling facilityexecutes the command to the point where a message response block (MRB)would be returned to the application. But, before sending the MRB andwhile the internal latches are held for the objects referenced by thecommand, a request is made to the signaling protocol engine in theprimary coupling facility to send a completion signal on the peer ISClink to the secondary coupling facility. Likewise, the cache/listcomponent of the secondary coupling facility executes the command to thepoint where the MRB would be returned to the application. But, beforesending the MRB and while the internal latches are held for the objectsreferenced by the command, a request is made to the signaling protocolengine in the secondary coupling facility to send a completion signal onthe peer ISC link to the primary coupling facility.

Step 4. The signaling protocol engine in the primary coupling facilitysends the completion signal to the secondary coupling facility and thenwaits for the reception of the completion signal from the secondarycoupling facility. Likewise, the signaling protocol engine in thesecondary coupling facility sends the completion signal to the primarycoupling facility and then waits for the reception of the completionsignal from the primary coupling facility.

Step 5. When the primary coupling facility recognizes the reception ofthe completion signal from the secondary coupling facility, the primarycoupling facility sends the MRB and releases the latches. Likewise, whenthe secondary coupling facility recognizes the reception of thecompletion signal from the primary coupling facility, it also sends theMRB and releases the latches. If a failure occurs during this period oftime and either the primary coupling facility or the secondary couplingfacility fails to recognize the reception of a completion signal, thenduplexing is broken by the coupling facility by resetting the duplexingactive indicator for the structure. (This is described in more detail ina subsequent section.)

Step 6. Assuming no errors have occurred, the LFSS receives both MRBsfrom the two coupling facilities and constructs a single messageresponse block by reconciling the results of the two MRBs and gives thisresponse to the application. If, on the other hand, duplexing has beenbroken by one of the two coupling facilities, then the operating systemwill invoke fail-over recovery and one of the two structures will beselected as the surviving instance. Once the error is corrected,duplexing can be reestablished.

User transparency is satisfied because the duplexing functions areperformed by the LFSS without awareness by the user application.

Failure isolation is satisfied by creating two copies of the structurein separate facilities, each of which can continue as the survivingstructure in a situation involving the failure of the other.

Command atomicity is satisfied by maintaining latches on both structuresuntil both commands complete.

Performance is optimized in several ways. First, sending the commands inparallel allows for maximum overlap of data transfer and commandexecution. Second, by exchanging completion signals immediately uponreaching the MRB send point in the command, the completion can bedetected with minimal intervening latency. Third, the amount of datasent in the signal itself is small relative to the amount of data senton the primary link for the command. So, a single peer ISC link canhandle the combined signal traffic generated by commands sent on asignificant number of primary ISC links. In fact, for small distances, asingle ISC link can handle the combined traffic of the commandsgenerated in a 32-system Parallel Sysplex. Fourth, by using listnotification as the signaling transport mechanism, the signal can beprocessed by the receiver channel engine without needing to interruptthe coupling facility control code (CFCC) to process the signal. Fifth,by using the SCIS facility, contention detected by a SEND MESSAGE can beminimized by causing redrives to be performed substantially immediately.

Although in the embodiment described above, an ISC link is used tocouple the two coupling facilities, this is only one example. Otherlinks may be used, including, for instance, an ICB or IC link. Further,more than two coupling facilities may be coupled to one another.However, it is not necessary for all coupling facilities to be coupledto each other. For instance, a third coupling facility may be coupled toCoupling Facility 2 via a peer connection, but not to Coupling Facility1.

In addition to the above, the coupling facilities that are coupled maybe in separate Central Processing Complexes (CPC), in separatepartitions of one CPC, or a combination thereof. In the situation thatthe facilities are in separate partitions of one CPC, the same links canbe used for both duplexing and command traffic.

In a further embodiment, multiple peer links can be configured asredundant connections. In this scenario, the process for sendingduplexing signals on the peer links (described herein) recognizes a linkfailure and maintains signal exchanges on surviving links.

(2) Alternative Model—Duplexing with Explicit Locks:

FIG. 2 depicts one embodiment a configuration 200, which includes twocoupling facilities 202, 204 coupled to a system 206 in a ParallelSysplex. The system is running an instance of the z/OS operating system208 and an application 210 that is coupled to a coupling facilitystructure 212 (either of cache or list type), whose location is notknown to the application. The actual physical connection is managed bythe LFSS component 214 of the z/OS operating system and commandsinitiated by the user application flow through the LFSS component.

Two instances of the coupling facility structure are maintained inseparate coupling facilities, referred to as the primary couplingfacility and the secondary coupling facility. In addition, a lockingstructure 216 owned and managed by LFSS is maintained in the primarycoupling facility.

One embodiment of the steps in a normal command execution for theexplicit locking alternative are shown in FIG. 2 in numeric sequencethat approximate the time sequence of the command. Each of these stepsis described below.

Step 1. The user application generates a command and communicates thiscommand to the LFSS through the system macro interface.

Step 2. A lock-and-record command is generated by LFSS and sent to thelocking structure in the primary coupling facility. A link-subsystem(LSS) component 218 in the primary coupling facility receives thecommand and transfers control to the list component which controls thelocking function. Information recorded in the locking structure includesthe MCB for the user command. The lock set by the command serializesaccess by other commands to the primary structure. (A modification wouldallow for a partitioning of the structure and the lock to cover thepartition of the structure impacted by the user command.)

Step 3. The lock-and-record command is executed by the primary couplingfacility and the MRB is returned to the LFSS. In this path, the lock issuccessfully obtained by the LFSS.

Step 4. The LFSS creates two copies of the command, sending one to theprimary coupling facility and the second to the secondary couplingfacility. The LFSS uses the asynchronous SEND MESSAGE interface withoutnotification to allow the two commands to be initiated in parallel. Theprotection provided by the lock in the locking structure enables the twocommands to be executed independently in the two coupling facilities.Parallel execution is preferred to minimize the elapsed time forexecuting the commands. The link-subsystem (LSS) component in theprimary coupling facility receives the command and transfers control tothe cache or list component, as appropriate. Likewise, thelink-subsystem (LSS) component in the secondary coupling facilityreceives the command and transfers control to the cache or listcomponent, as appropriate.

Step 5. Each coupling facility independently executes its copy of theuser command and returns the result in the MRB. The two MRBs returnedare identical. (Any differences that cannot be corrected by retrying thecommands result in loss of duplexing mode.)

Step 6. The MRB is returned to the user application. From the user'sperspective the command has been completed.

Step 7. The LFSS generates an unlock-and-clear command to the lockingstructure to free the lock and delete the recorded information.

Step 8. The primary coupling facility executes the locking command andreturns the MRB indicating success.

This ends the normal path for this alternative duplexing design.However, in step 3, the lock may already be held when the lockingcommand is executed. In this case, the lock operation fails with anon-zero response code and the record operation is suppressed. Thefollowing sequence of steps occurs beginning at step 3.

Step 3′. The locking command is executed unsuccessfully by the primarycoupling facility. The lock is already held by another system. In thiscase, the record operation is not performed and a response code isgenerated that indicates ‘lock held’. The MRB containing this responseis returned to the LFSS.

Step 4′. The LFSS reissues the lock-and-record command to the primarycoupling facility.

Step 5′. The primary coupling facility executes the locking commandeither successfully or unsuccessfully. The result is returned in the MRBwith a response code. If execution is successful, the processing resumesat step 4 above. If execution is unsuccessful, the locking command isreissued at step 4′. This continues for a model-dependent time period bythe LFSS, after which duplexing is broken.

User transparency is satisfied because the locking and duplexingfunctions are performed by the LESS without awareness by the userapplication.

Failure isolation is satisfied by creating two copies of the structurein separate facilities, each of which can continue as the survivingstructure in a situation involving the failure of the other. The lockingstructure is only used for duplexing purposes, and so it can bediscarded by the recovery program performing structure fail-over.

Command atomicity is satisfied because all accesses to the primary andsecondary structures are serialized by the explicit lock operation.

One shortcoming of this approach is the performance cost of theadditional lock and unlock commands. These are not overlapped with themainline commands. In addition to this shortcoming, there are severalother performance concerns. First, there is a significant additionalload placed on the links to the coupling facility containing the lockstructure due to the generation of a pair of lock-and-record andunlock-and-clear commands for each user generated command. Also, theexplicit serialization introduced at the structure level or at apartition of the structure introduces new contention that will blockcommand execution. The current design of using internal latching in thecoupling facility which optimizes concurrent execution is defeated bythis approach.

(3) Alternative B—Store and Forward Duplexing

FIG. 3 depicts one embodiment of a configuration 300 including twocoupling facilities 302, 304 in a Parallel Sysplex. The primary couplingfacility is connected to a system 306 in the Sysplex that is running aninstance of the z/OS operating system 308 and an application 310 that iscoupled to a coupling facility structure 312 (either of cache or listtype), whose location is not known to the application. The actualphysical connection is managed by the LFSS component of the z/OSoperating system and commands initiated by the user application flowthrough the LFSS component. The secondary coupling facility is coupledto the primary coupling facility by a standard ISC link 314 with thesender side connected to the primary coupling facility and the receiverside connected to the secondary coupling facility. The secondarycoupling facility is also connected to the system, but the connection isnot used for duplexing operations. The connection is used in the eventof a structure failure where the secondary structure is selected. So,the connection is depicted in dashed lines to represent its state as apassive backup connection.

One embodiment of the steps in a normal command execution for thestore-and-forward alternative are shown in FIG. 3 in numeric sequencethat approximates the time sequence of the command.

Step 1. The user application generates a command and communicates thiscommand to the LFSS through the system macro interface.

Step 2. The LFSS issues the command to the primary coupling facility. Alink-subsystem (LSS) component 316 in the primary coupling facilityreceives the command and transfers control to the cache or listcomponent, as appropriate.

Step 3. The cache/list component of the primary coupling facilityexecutes the command to the point where the MRB would be returned to theapplication. However, before sending the MRB and while internal latchesare held for the objects referenced by the command, a request is made toa command-forwarding subsystem 318 to forward the user command to thesecondary coupling facility.

Step 4. The command-forwarding subsystem in the primary couplingfacility issues a standard SEND MESSAGE instruction to send the commandon the ISC link connected to the secondary coupling facility and waitson the MRB.

Step 5. The command forwarding component in the secondary couplingfacility receives the command from the primary facility and transferscontrol to the cache or list component, as appropriate.

Step 6. The cache/list component of the secondary coupling facilityexecutes the command against the secondary structure and hands the MRBto the command forwarding component.

Step 7. The command forwarding component in the secondary couplingfacility sends the MRB to the command forwarding component in theprimary coupling facility. Command execution is complete in thesecondary coupling facility.

Step 8. The command forwarding component in the primary couplingfacility receives the MRB from the secondary coupling facility andtransfers control to the cache/list component.

Step 9. The cache/list component determines the command execution in thesecondary coupling facility is successful and returns the primary MRB tothe LFSS.

Step 10. The MRB is returned to the user application.

User transparency is satisfied because the duplexing functions areperformed by the coupling facilities without awareness by the userapplication.

Failure isolation is satisfied by creating two copies of the structurein separate facilities, each of which can continue as the survivingstructure in a situation involving the failure of the other.

Command atomicity is satisfied because accesses to the primary structureare serialized, while the command is forwarded to the secondary couplingfacility.

There are shortcomings with this model, however. For instance, commandsto the secondary structure are to pass over the peer ISC link, includingthe data transfers. This creates a funneling effect between the set ofprimary ISC links connected to the primary coupling facility from thesystems in the Sysplex to the single peer ISC link, which creates abottleneck in the system design. To overcome this, multiple links are tobe established as peer ISC links to handle this combined traffic. Thisraises the expense of the configuration and reduces the connectivity ofthe coupling facilities to systems. Another problem is that the primaryand secondary commands execute serially, with little opportunity foroverlapping either command execution or data transfer. A third problemis that the peer ISC link is to be fully functional, including datatransfers and timeouts. The peer ISC link for the couplingfacility-to-coupling facility model only transfers simple messageexchanges during the configuration phase and only transfers secondarycommands during command processing. This significantly lowers thecomplexity of the link support in the CFCC, compared to thefull-function support needed in the store-and-forward model.

Coupling-Facility Connections

A coupling facility is attached to a second coupling facility when acoupling facility link exists from the first facility to the secondfacility and the second facility is in the managed state. In this case,we say that the second coupling facility is remotely attached to thefirst coupling facility. A coupling facility may have any number ofremotely attached coupling facilities and may have multiple couplingfacility links to the same coupling facility.

The attachment to a remote facility is active, when at least one messagepath to the remote facility is placed in the active state by anActivate-Message-Path (AMP) command (described hereinafter). In oneexample, attachments are not activated, unless the coupling facility isin the managed state and a signaling vector is defined.

Two coupling facilities are connected if each coupling facility isremotely attached to the other by active attachments.

Connected coupling facilities can be used to support many functions,including, for instance the duplexing of structures. The couplingfacility links that enable the connection are used to transport signalsto remote signaling vectors (described hereinafter) in the attachedfacilities. The program (e.g., LFSS) issues commands to the couplingfacilities containing the two structures, and a sequence of signals areexchanged between the two coupling facilities as part of commandexecution. The sequence of signals, referred to as the signalingprotocol, is used to coordinate the updates to the structure objects sothat the pair of structures can be synchronized. The signaling protocolextends the command atomicity rules across the pair of structures,ensuring that updates to the set of objects referenced by the commandare either applied or suppressed as a single atomic operation in bothcoupling facilities.

A pair of structures that is synchronized across a coupling facilityconnection is called a duplexed pair of structures, and each of the twostructures is in the duplexing-active state. An error condition orconfiguration action may cause the duplexing state to be broken. Whenduplexing is broken, one of the two structures is chosen to continue tosupport command execution against the objects in the structure. Thesurviving structure is said to be executing in the simplex state. Theother structure no longer accepts commands and is deallocated by theprogram. If a copy of the structure running in simplex state is createdon a connected coupling facility, the duplexing state can bereestablished between the active structure and the copied structure.

FIG. 4 depicts an example configuration 400 of a collection of couplingfacilities. In FIG. 4, System A, which is one of potentially manysystems in a Parallel Sysplex, is attached to three coupling facilities:System A is attached to Coupling Facility y via sender ISC Channel (S)1, to Coupling Facility x via sender ISC Channel 2, and to CouplingFacility z via sender ISC Channel 3. All of these attachments are in theactive state. The association of sender ISCs to coupling facility's andtheir attached state are reflected in a configuration table 402 in thez/OS operating system. Other systems in the Parallel Sysplex may also beattached to Coupling Facilities x, y and z and their configurationtables would reflect the state of their attachments.

In addition, the coupling facilities are attached to each other, butthey are not fully interconnected. Coupling Facility x is attached toCoupling Facility z by sender ISC Channel 2, but Coupling Facility x isnot attached to Coupling Facility y, in this example. This is reflectedin the coupling facility x configuration table, which shows a singleactive (A) attachment for sender ISC 2 to Coupling Facility z. Likewise,Coupling Facility y is attached to Coupling Facility z by sender ISC 7.Coupling Facility z is attached to Coupling Facility y, but not toCoupling Facility x. In fact, Coupling Facility z has two attachments toCoupling Facility y, sender ISC 3 and 6. However, while sender ISC 3 isan active attachment, the attachment via sender ISC 6 has not yet beenactivated (I).

This collection of attachments generates one pair of connected couplingfacilities. Namely, the pair of ISC links represented by sender ISC 7 inCoupling Facility y and sender ISC 3 in Coupling Facility z are activeattachments in each direction, and therefore, form a peer ISC linkbetween Coupling Facility y and Coupling Facility z. Therefore, CouplingFacility y and Coupling Facility z are connected coupling facilities. Nosuch connection exists between Coupling Facility x and Coupling Facilityy or between Coupling Facility x and Coupling Facility z. So, astructure is duplexed by placing one structure in Coupling Facility yand the other structure in Coupling Facility z.

A program running on System A can allocate a duplexed pair ofstructures: one in Coupling Facility y and one in Coupling Facility z.If at some point in time, Coupling Facility y fails, or one of the ISClinks in the peer ISC link attaching Coupling Facility z to CouplingFacility y fails, the duplexing-active state can be broken by theprogram and the surviving structure on one of Coupling Facility y orCoupling Facility z can be selected and accessed as a single structureand the structure executes in a simplex state. The change from duplexstate to simplex state is done without loss of data and with minimalrecovery time required for switching states. Assuming the survivingstructure is on Coupling Facility z and later a second copy of thestructure is created on Coupling Facility y, the structure can reenterthe duplexed state. If, in the interim, an ISC link is established fromCoupling Facility z to Coupling Facility x and attachments in bothdirections are activated, the second copy of the structure could beplaced on Coupling Facility x and the signaling protocol would flowbetween Coupling Facility z and Coupling Facility x. Likewise, ifCoupling Facility z or it's corresponding ISC link fails, then thestructure in Coupling Facility y is the surviving structure and isaccessed in simplex mode. In this case, however, the lack of a signalingchannel between Coupling Facility x and Coupling Facility y preventsreentering the duplexed state until Coupling Facility z is repaired, oruntil ISC links are configured between Coupling Facility x and CouplingFacility y in both directions.

The coupling facility configuration tables reflect the states of each ofthe ISC links associated with the sender ISC channels. A similar tableexists for the receiver ISC channel, called the Message-Path StatusVector (MPSV). This is an architected table in the coupling facilitythat is extended for duplexing connections. The MPSV and its extensionsare described in the next section. The CFCC determines the collection ofconnected coupling facilities from information in both the configurationtable and the MPSV. A source node descriptor in the MPSV is theinformation that links the sender and receiver ISC channels.

Message-Path Objects for Coupling Facility Duplexing

In general, several processors issue coupling-facility commands, andseveral links may connect each processor to the coupling facility. Thecoupling facility assigns message-path identifiers that distinguishamong the different sources of commands. Further, associated with amessage path are various objects, which are used to, for instance,define the path, provide status, etc.

Examples of message-path objects are summarized in the following tableand described below. There may be more, less or different message-pathobjects.

Message-Path Objects Acronym Image identification code IID Message-pathidentifier MPID Message-path request level MPRL Message-path state MPSSource node descriptor SRCND System identifier SYID Path groupMessage-path status vector

Image Identification Code (IID): A value that specifies the imageassociated with the message path. The object is initialized to zero andis set to the value of the IID field in the message header, when anActivate-Message-Path command (described hereinafter) is processed.

Message-Path Identifier (MPID): A value that is used to identify amessage path at the coupling facility. There is a unique message-pathidentifier for each source of direct or intermediate commands. Thevalues of the message-path identifiers are set by an installationprocedure. They are not modified by any command.

Message-Path Request Level (MPRL): A value associated with each messagepath that indicates the maximum number of commands that may be processedconcurrently for the message path. The value is initialized when themessage path becomes operational and remains unchanged, while the pathis in the operational state.

Message-Path State (MPS): A value that specifies the state of themessage path. A value of X′00′ indicates the path is in the inactivestate, and a value of X′01′ indicates the path is in the active state.

Source Node Descriptor (SRCND): An optional value that is designated bythe program when the message path is activated, which contains thenode-descriptor object of the message path source. When a message pathis activated and a source node descriptor is not provided, thesource-node-descriptor object is in the empty state.

System Identifier (SYID): A value that is designated by the program whenthe message path is activated.

Path Group: The set of message paths with the same system-identifier(SYID) value for message paths that do not have source node descriptors.For the message paths that have source node descriptors, the path groupincludes the set of message paths with the same system-identifier valueand source-node-descriptor value.

Message-Path Status Vector: A message-path-status-vector object includesstate information for each message path in the coupling facility. Themessage-path status vector includes, for instance, the following objectsfor each message path:

The message-path state (MPS);

The message-path request level (MPRL);

The source node descriptor (SRCND) (optional);

The system identifier (SYID); and

The image identification code (IID).

After coupling facility initialization is complete, the state of eachmessage path is inactive.

One example of a message-path status vector is depicted in FIG. 5. Asshown, the fields include, for instance, MPS, MPRL, SYID and the IIDfield. It also includes a column for the source node descriptor, whichis a field used for duplexing. The field is set by an Activate MessagePath command under control of the SRCNDC operand. (See the section onmessage-path operands and message-path commands below.) The source nodedescriptor field is marked as empty for message paths originating withinan operating system partition and is set to the value of the nodedescriptor object for paths originating in remote coupling facilities.The source node descriptor object is used by the CFCC to match couplingfacility receiver chpids (CFR) and coupling facility sender chpids(CFS). Activating a message path to a remote coupling facility indicatesthat the remote coupling facility is attached. If there is also anactive message path in the MPSV for the same remote coupling facility(i.e., the source node descriptor is equal to the remote couplingfacility node descriptor), then the two coupling facilities areconnected.

Global Objects

In addition to message path objects, global objects resident within thecoupling facility are used to identify the coupling facility, describeits state, define its model-dependent limitations and summarize thestatus of its resources. Global objects include, for instance, fixedglobal controls, which are set at coupling facility power-on reset andare not modified by any command or coupling facility process; andprogram-modifiable global controls, which are initialized at couplingfacility power-on reset and may be modified by subsequent commands orcoupling facility processes. Although various global objects aredescribed below, more, less or different objects may exist.

One embodiment of a fixed global control used for duplexing issummarized in the following table and described below.

Fixed Global Controls Acronym Signaling-vector token SVT

Signaling-Vector Token (SVT): A value that identifies the signalingvector. When the signaling-vector token is zero, a signaling vector isnot provided in the coupling facility.

Likewise, examples of program-modifiable global controls are summarizedin the following table and described below.

Program-Modifiable Global Controls Acronym Authority AUChannel-connection controls Duplexing vector DV Remote-facilities tableRFT Remote-facilities-access counter RFAC Remote-facility controls Retryvector SID vector Signal group SV Signaling controls SGP Signalingvector SGNLV

Authority (AU): A program-designated coupling-facility-state value. Whenthe value is non-zero, the coupling facility commands are executednormally. When the value is zero, the execution of certain couplingfacility commands is suppressed. The authority value is initialized tozeros. The authority contains a sub-object, called the system identifier(SYID).

Channel-Connection Controls: One set of channel-connection controlsexists for a coupling facility. Examples of channel-connection controlsare summarized in the following table and described below.

Channel-Connection Controls Acronym Count of Installed and Not CINOPCOperational Channels Inst. And Not Oper. Chan. INOPCP Path Descriptors

Count of Installed and Not Operational Intersystem Channels (CINOPC): Avalue that includes the number of channel-path identifiers assigned tointersystem channels that are installed in the coupling facility and arein the not operational state. The value of the count of installed andnot operational intersystem channels is, for instance, between zero andsixty four.

Installed and Not Operational Channel-Path Descriptors (INOPCP): Anordered collection of values that include the channel-path-identifier(CHPID) values that are installed in the coupling facility and are notoperational. The number of installed and not operational channel pathdescriptors is included in the CINOPC control. The installed and notoperational channel paths are ordered by type and by value. The senderchannel paths are listed first followed by the receiver channel paths.Within each type, the channel-path identifiers are ordered from lowestto highest. When the number of installed and not operational channelpaths exceeds 64, then the list is truncated and the first 64channel-path identifiers are returned.

Duplexing Vector (DV): A bit string with an initial value of zero. Thebit positions start at 0 and increase sequentially to the SID limit. Thebit at position (i) in the string is set to one, when a structure ismade duplexing active with a SID value of (i). The bit at position (i)is reset to zero when duplexing is broken by the CFCC or explicitlydeactivated by the OS, or the structure is deallocated. All bits in theduplexing vector are reset to zero when the authority object is changedby a Set-Facility-Authority command and a preserve-duplexing indicator(described hereinafter) is zero, or when a coupling facility power-onreset occurs. A bit in the duplexing vector is referred to as theduplexing-active bit for the associated structure.

In one aspect, the obtaining of this vector includes, but is not limitedto, defining the vector, receiving the vector or otherwise, beingprovided the vector.

Remote-Facilities Table (RFT): Information that is returned as a resultof a Read Connected Facility Controls command (described hereinafter).

Remote-Facilities-Access Counter (RFAC): A value that is incremented,whenever a remote coupling facility's accessibility level changes eitherfrom not connected to connected or from connected to not connected. Thevalue is set to zero, when a coupling facility power-on reset occurs.

Remote-Facility Controls:

One or more remote coupling facilities may be accessible by messagepaths between the facilities. There are two possible levels ofaccessibility to a remote facility:

Not connected;

Connected.

The remote facility is connected, when at least one message path to theremote facility is in the active state and at least one message pathfrom the remote facility is in the active state, as recorded in themessage-path status vector. Otherwise, the remote facility is notconnected.

As examples, a remote facility may be not connected for any of thefollowing reasons:

-   -   No message paths exist to the remote coupling facility. It may        be the case that no message path ever existed to the remote        facility, or that the coupling facility had previously been        accessible, but all message paths have been removed.    -   All the message paths to the remote facility are inactive. This        state may exist for several reasons: (1) The coupling facility        link has been varied online and the coupling facility has not        yet activated the message path; (2) the remote facility is not        in the managed state or does not possess a signaling vector;        or (3) a Deactivate Message Path (DMP) command (described        hereinafter) has been issued on the last active message path to        the remote facility.    -   At least one message path to the remote facility is in the        active state, but no active message paths from the remote        facility exist.    -   The remote-facility node descriptor matches the node-descriptor        object. This ensures that a coupling facility cannot be        connected to itself as a remote facility.

A set of remote facility controls exists for each recognized remotefacility.

The remote-facility controls are created when an Identify Message Pathcommand (described hereinafter) completes and the node descriptor andsystem identifier that are returned do not match any existing remotefacilities.

Examples of remote-facility controls are summarized in the followingtable and described below.

Remote-Facility Controls Acronym Remote-facility accessibility levelRFAL Remote-facility channel path identifiers RFCHPRemote-facility-controls time of creation RFCTOC Remote-facilitydisconnect time RFDT Remote-facility disconnect time validity RFDTVIindicator Remote-facility node descriptor RFND Remote-facility pathdescriptors RFPD Remote-facility path-group size RFPGS Remote-facilitysender channel-path identifiers RFSCHP Remote-facility sender pathdescriptors RFSPD Remote-facility sender path group size RFSPGSRemote-facility signaling-vector token RFSVT Remote-facility systemidentifier RFSYID Remote-facility signaling counters

Remote-Facility Accessibility Level (RFAL): A value that describes thecurrent level of accessibility to the remote coupling facility. It hasthe following encoding:

0 Not connected; 1 Connected.

Remote-Facility Channel-Path Identifiers (RFCHP): An ordered collectionof values that include the channel-path-identifier (CHPID) valuesassigned to the message paths in the remote-facility path group. Thereis a remote-facility channel-path identifier for each active messagepath in the path group, and therefore, the number of remote-facilitychannel-path identifiers is equal to the remote-facility path-group size(RFPGS). The ordering of the remote-facility channel-path identifiersmatches the ordering of the remote-facility path descriptors, i.e., theith remote-facility path descriptor includes the value of the descriptorfield returned by the store-channel-path-descriptor command of theChannel Subsystem Call (CHSC) instruction when the ith remote-facilitychannel is specified as the input operand.

Remote-Facility-Controls Time of Creation (RFCTOC): A time-of-day (TOD)value indicating the time when the remote facility controls are created.

Remote-Facility Disconnect Time (RFDT): A value that includes theelapsed time that has occurred since the remote facility changed fromconnected to not connected. The format of the remote-facility disconnecttime matches the S/390 time-of-day clock. For example, bit 51corresponds to 1 millisecond.

Remote-Facility Disconnect Time Validity Indicator (RFDTVI): A valuethat determines the validity of the remote-facility disconnect timeobject. It has the following encoding:

0 Value of the RFDT object is not valid; 1 Value of the RFDT object isvalid.

Remote-Facility Node Descriptor (RFND): A value that includes thenode-descriptor object of the remote coupling facility.

Remote-Facility Path Descriptors (RFPD): A collection of values thatdefine the channel-path types for the channel paths in the path groupassociated with the remote facility. There is a remote-facility pathdescriptor for each active message path in the path group, and thereby,the number of remote-facility path descriptors is equal to theremote-facility path-group size (RFPGS).

Remote-Facility Path-Group Size (RFPGS): A value that includes thenumber of active message paths in the path group associated with theremote facility. The value of the remote-facility path-group size isbetween zero and eight, as one example.

Remote-Facility Sender-Channel-Path Identifiers (RFSCHP): An orderedcollection of values that include the channel-path-identifier (CHPID)values of the sender intersystem channels that are connected to theremote facility. A sender intersystem channel is connected to a remotefacility, when the channel is operational and an Identify-Message-Pathcommand issued on the channel path returns the node-descriptor objectassociated with the remote facility. There is a remote-facility senderchannel-path for each connected sender intersystem channel, andtherefore, the number of remote-facility sender-channel-path identifiersis equal to the remote-facility sender-path-group size (RFSPGS). Theordering of the remote-facility sender-channel-path identifiers matchesthe ordering of the remote-facility sender path descriptors; i.e., theith remote-facility path descriptor contains the value of the descriptorfield returned by the store-channel-path-description CHSC command whenthe ith remote-facility sender-channel-path identifier is specified asthe input operand.

Remote-Facility Sender Path Descriptors (RFSPD): An ordered collectionof values that define the channel-path types for the sender intersystemchannels that are connected to the remote facility. There is aremote-facility sender path descriptor for each sender intersystemchannel that is connected to the remote facility, and thus, the numberof remote-facility sender path descriptors is equal to theremote-facility sender-path-group size (RFSPGS). The value of theremote-facility sender path descriptor is set to the value of thedescriptor field (DESC) for the associated channel-path type returned bythe store-channel-path-description CHSC command.

Remote-Facility Sender-Path-Group Size (RFSPGS): A value that includesthe number of sender intersystem channel that are operational andconnected to the remote facility. The value of the remote-facilitysender-path-group size is between, for instance, zero and eight.

Remote-Facility Signaling-Vector Token (RFSVT): A value used to identifythe signaling vector in the remote coupling facility.

Remote-Facility System Identifier (RFSYID): A value that is designatedby the program when the remote coupling facility is placed in themanaged state.

Remote-Facility Signal Counters: A set of remote-facility signalcounters is associated with each remote facility. The signal countersare initialized to zero when the remote-facility controls are created.The signal counters are defined as substantially accurate.

Examples of remote-facility signal counters are summarized in thefollowing table and described below.

Remote-Facility Signal Counters Acronym Delayed signal counter DSCHalt-execution signal counter HESC Ready-to-complete signal counterRTCSC Ready-to-execute signal counter RTXSC Request-for-suppressionsignal counter RFSSC Request-for-suppression-accepted signal counterRFSASC Signal delay time first moment SDTFM Signal delay time secondmoment SDTSM Signal-redrives signal counter SRDSC Signal service timefirst moment SSTFM Signal service time second moment SSTSM

Delayed-Signal Counter (DSC): A value that indicates the number ofsignals delayed in being sent to the remote facility.

Halt-Execution Signal Counter (HESC): A value that indicates the numberof halt-execution signals sent to the remote facility.

Ready-to-Complete Signal Counter (RTCSC): A value that indicates thenumber of ready-to-complete signals sent to the remote facility.

Ready-to-Execute Signal Counter (RTXSC): A value that indicates thenumber of ready-to-execute signals sent to the remote facility.

Request-for-Suppression Signal Counter (RFSSC): A value that indicatesthe number of request-for-suppression signals sent to the remotefacility.

Request-for-Suppression-Accepted Signal Counter (RFSASC): A value thatindicates the number of request-for-suppression-accepted signals sent tothe remote facility.

Signal Delay Time First Moment (SDTFM): A value that includes theaccumulated delay time in, for instance, microseconds for signalsdelayed in being sent to the remote facility.

Signal Delay Time Second Moment (SDTSM): A value that includes theaccumulated squares of delay time in squared-microsecond units forsignals delayed in being sent to the remote facility.

Signal-Redrives Signal Counter (SRDSC): A value that indicates the totalnumber of redrives of signals to the remote facility.

Signal Service Time First Moment (SSTFM): A value that includes theaccumulated service time, excluding delay time, in microseconds forsignals sent to the remote facility.

Signal Service Time Second Moment (SSTSM): A value that contains theaccumulated squares of service time, excluding delay time, insquared-microsecond units for signals sent to the remote facility.

Notes on Remote Facility Controls:

-   -   1. If the counters are sampled at periodic intervals, the        sampling program verifies that the counters have not been reset        during the sampling interval. This can occur if the remote        facility becomes disconnected, the controls released and then        subsequently reacquired. The program can detect this by storing        the remote-facility-controls time of creation and comparing this        value each time the counters are read with the current value.    -   2. The signal counters include logical requests to deliver        signals, including signals that are never successfully        delivered. But, the signal counters do not include any redrives        of signals.    -   3. Redrives of signals are included in the first and second        moments of service time, provided at least one signal is        successfully delivered.    -   4. The signals delivered to a remote facility should be        distributed uniformly across the message paths in the path        group, through a scheduling technique, such as round-robin. This        allows for full utilization of link resources configured between        the facilities.    -   5. The remote-facility controls should not be released until the        last sender channel is configured to a different remote        facility. If the remote facility is disconnected, but at least        one sender channel remains associated with the path group, then        the possibility exists that the connection may be reestablished        and the controls should be preserved. This minimizes the effect        of resetting the signal counters in the presence of transient        link errors.    -   6. The remote-facility disconnect time can be determined by        storing a time stamp in the controls associated with a sender        ISC whenever the corresponding message path becomes inactive.        The disconnect time value can be calculated when a Test Remote        Facility Access (TRFA) command (described hereinafter) is        executed by subtracting the stored time stamp from the current        Time of Day (TOD) value. If more than one path exists in the        path group to the remote facility, then the disconnect time is        calculated when all the message paths are inactive. In this        case, the value of the disconnect time is calculated using the        most recent message path to become inactive.    -   7. If a message path is reactivated for a sender ISC, but the        new remote facility is different from the previous remote        facility, then the time stamp should be reset. If this is the        last message path in the path group to the old remote facility,        then the remote-facility-disconnect-time validity indicator        should be set to zero.    -   8. The remote-facility disconnect time is returned by a Test        Remote Facility Access (TRFA) command (described hereinafter),        when the remote-facility is disconnected. The program can limit        the spin time for waiting for a lost connection to be restored        by using the RFDT value to bound the spin rather than a locally        maintained time value.    -   9. The count of installed and not operational intersystem        channels includes both sender and receiver ISCs. If the channel        is a coupling-facility peer channel (CFP), then the channel may        appear to be installed twice, once for the sender side of the        channel and once for the receiver side. If a CFP channel is        installed in a coupling facility as both a sender channel and a        receiver channel and is recognized as not operational, the count        of installed and not operational intersystem channels is        increased by two.    -   10. A receiver intersystem channel may be connected to a sender        intersystem channel that is shared by two or more remote        coupling facilities. So, the same CHPID value may appear in        multiple RFCHP controls.    -   11. A sender intersystem channel may be connected to a receiver        intersystem channel that is shared by two or more remote        coupling facilities. So, the same CHPID value may appear in        multiple RFSCHP controls.    -   12. The remote facility path group has a message path for each        receiver channel over which it has processed an AMP command from        the remote facility and for which the paths are still active.        The CHPID values for the receiver channels associated with the        path group are returned in the RFCHP object, the corresponding        path descriptors are returned in the RFPD object, and the count        of receiver channels is returned in the RFPGS object.

Referring back to the program-modifiable global controls table above,additional controls exist, which are described below.

Retry Vector: A vector that includes retry-index (RX) values for theretry buffers of a coupling facility.

SID Vector: A vector that includes structure identifier (SID) values forthe structures of a coupling facility.

Signal Group (SGP): A collection of signal values associated with theexecution of a command or a portion of a command. Each signal groupincludes, for instance, the following signals:

RTE Ready To Execute The duplexed command is ready to begin execution.RTC Ready To Complete The duplexed command has completed execution ofthe command or a portion of the command and is ready to commitcompletion status. HE Halt Execution The duplexed command hasencountered an asymmetric resource condition, including, for instance, atimeout condition, a latch contention condition, a storage constraint,or failure of a conditional test request. The command execution is tostop at the completion of the previous command portion, or lacking aprevious portion, the command is suppressed. RFS Request For SuppressionA command-sequence-number conflict has occurred for the duplexed commandand command suppression is requested. RFSA Request For Suppression Theduplexed command has Accepted accepted an RFS signal. The commandexecution is to stop at the completion of the previous command portion,or lacking a previous portion, the command is suppressed.

When a signal is received for a signal group, the signal is set to oneand remains in this state until the signal group is reset. When thesignal group is reset, each signal is set to zero.

Signaling Controls:

One example of various signaling controls are summarized in thefollowing table and described below.

Signaling Controls Acronym Command sequence number CSNCurrent-signal-group index CSGX

Command Sequence Number (CSN): A number associated with a currentlyexecuting duplexed command. The initial value is zero.

Current Signal-Group Index (CSGX): A value that identifies the currentlyactive signal group in a signaling vector entry (described below). Theinitial value is B′01′, as one example.

Signaling Vector (SGNLV): The signaling-vector object includes a lineararray of signaling-vector entries, where each entry includes signalingcontrols and a plurality of (e.g., three) signal groups, as depicted inFIG. 6. Each entry 600 is associated with a retry index (RX) 602, whichis the index into the signaling vector, and each entry includes twosignaling controls 604: the command sequence number and the currentsignal-group index, and three signal groups 606. The reset and initialstate of a signaling-vector entry are the same. When an entry is reset,the command sequence number is set to zero, the current signal-groupindex is set to one, and each signal group is reset. After couplingfacility initialization is complete or after the authority object ischanged by, for instance, a Set-Facility-Authority command (describedhereinafter) and the preserve-duplexing indicator is zero, eachsignaling-vector entry is in the reset state.

In one example, the signaling vector is to be large enough to map eachretry buffer available in the coupling facility. If a vector ofsufficient size cannot be created, then no signaling vector isestablished and no remote facility connections are made.

Notes on Signaling Controls:

-   -   1. The signaling vector is created, for instance, by a DEFINE        VECTOR instruction, using the list-notification option. The CFCC        creates the vector when the CFCC initializes. The        list-notification token returned by the instruction is placed in        the signaling-vector-token object. In one example, Bit 17 is set        to B′1′, when the token is assigned. Thus, the token returned by        the DEFINE VECTOR instruction is non-zero.    -   2. Signals are implemented as list-notification commands using        the list-notification token to identify the signaling vector in        the neighboring facility. The semantic content of the signal is        coded in the list-notification entry number.    -   3. The synchronization protocol for duplexing list-form commands        employs, for instance, three signal groups to be provided for        each signaling-vector entry. This allows signals to be received        on the previous, current, or next list item that is processed.    -   4. The duplexing state of a structure may be changed from the        active state to the inactive state, when the coupling facility        detects a loss of connectivity that may result in the loss of        one or more signals sent between the paired coupling facilities.        Once the duplexing state is set to inactive, duplexing can only        be reactivated by the program.    -   5. A message path from each remote coupling facility is        activated by means of an Activate-Message-Path (AMP) command        sent on the coupling link to the remote facility. The SYID        request operand in the AMP command is set equal to the value of        the system identifier in the authority object.    -   6. A coupling facility cannot be included in its own list of        remote facilities. Duplexing is only activated, in this example,        when the two structures reside in distinct coupling facilities.        This is enforced by requiring that for a remote facility to be        connected, the RFND is not to be equal to the node-descriptor        object.    -   7. The maximum number of retry buffers that a coupling facility        can support with duplexing is limited by the maximum size vector        that can be defined for a logical partition. The LNEN format for        duplexing signals places the retry index in bits 9-24 (for        example), allowing for 128 bits per signaling vector entry. So        the retry index limit is not to exceed one less than the maximum        vector size divided by 128. On G6 machines, this limit is        ‘40000’X or 256 K bits. So, the retry index limit is not to        exceed 256K/128−1 or 2047. However, the coupling facility is to        provide at least as many retry buffers as there are recipient        channel buffers. This is for the OS/390 IFCC recovery        techniques. If the signaling vector size cannot support this        amount of retry buffers, then the retry index limit is not        changed and the signaling vector is not created. Remote facility        connections are not made, and hence, duplexing is not        established for structures in this facility.

This completes a description of one embodiment of various globalobjects. One or more global objects are used, in one example, along withone or more message path objects, to define a configuration. One suchconfiguration is described in FIG. 7.

As depicted in FIG. 7, a configuration 700 includes two couplingfacilities 701, 702 coupled to a single system 704 running the z/OSoperating system. In this configuration, the z/OS image has two senderchannels, CFS 2 and CFS 3. CFS 2 is connected to receiver channel CFR 5on CF 1, and CFS 3 is connected to receiver channel CFR 6 on CF 2. Inthis implementation, the MPID is set equal to the chpid number for thereceiver chpid. In CF 1, the message path originating in the z/OS imageis reflected in row 5 (706) of the MPSV 708. The state of the messagepath is ‘active’ (A), the MPRL is 2, the system id (SYID) is the systemid for z/OS, the image id is the partition number of the z/OS image andthe source node descriptor is empty. The entry in the MPSV in CF 2 forthe message path originating in CFS 3 is similar.

There are two additional message paths defined between the two couplingfacilities. The first originates in CF 1 in CFS 11 and is connected toCFR 9 in CF 2. Row 9 in the MPSV in CF 2 reflects the state of thismessage path. The path is active, which shows that CF 1 is attached toCF 2. The MPRL is 2, the system identifier is set to the value of thehigh order doubleword of the authority in CF 1, the IID is set to thepartition number for the logical partition that is running CF 1, and thesource node descriptor is set to the node descriptor for the systemcontaining CF 1. These fields are all established by the AMP commandissued by CF 1 to CF 2 over CFS 11.

A similar set of fields is set in the MPSV on CF 1 for the message pathoriginating in CFS 10 on CF 2 and connected to CFR 8 on CF 1. The twocoupling facilities are in the connected state and are eligible tocontain a duplexed pair of structures. Duplexing signals sent from CF 1to CF 2 use the secondary message buffers on CFR 8 via list notificationcommands. Likewise, CF 2 sends duplexing signals to CF 1 using thesecondary message buffers on CFR 9. The MPRL value of 2 indicates thattwo duplexing signals may be sent in parallel in each direction. TheCFCC monitors the state of the connection to allow duplexing to bemaintained by monitoring the state of both the CFR link and the CFSlink. In the case of CF 1, that means both CFR 8 and CFS 11 are to beoperational and active. Likewise, CF 2 monitors CFR 9 and CFS 10.

Retry-Buffer Objects

Other objects are also employed to support coupling facility processing.These objects include, for instance, retry-buffer objects, which aredescribed below.

A retry buffer is an area of coupling facility storage that includesinformation relevant to command recovery. The retry buffers are assignedby the LFSS component of z/OS when a coupling facility is initiallyconnected. The LFSS assigns a retry index within the range of retryindexes assigned to that system to a message subchannel that contains asender ISC channel, which is connected to the coupling facility. Theretry index identifies a unique collection of objects in the couplingfacility that constitute the retry buffer. These include, for instance,the retry authority, the retry information, the retry data block, and asignaling vector entry.

As depicted in FIG. 8, a retry buffer 800 includes a retry-informationportion 802, which is assigned from an area of coupling facility storagethat is not available for structure allocation; and a retry-data-blockportion 804, which is assigned from the structure-storage areaassociated with the retry data.

Each command that uses a retry buffer specifies a retry index (RX) as arequest operand. The coupling facility places the retry data into thespecified retry buffer as part of command execution. The retryinformation is returned in the message-response block and the retry-datablock is returned to the location specified by the data-block address inthe message-command block.

When the size of the data stored in the retry-data block is less thanthe maximum data size, the data is left justified in the retry-datablock. The length of the retry data is placed in the data-count field ofthe response header.

Examples of retry-buffer objects are summarized in the following tableand described below with reference to FIG. 8.

Retry-Buffer Objects Acronym Command-recovery information Previousresponse code PRC Previous status conditions PSC Previousduplexing-deactivated indicator PDDI Retry-buffer authority RBAU Retryversion number RVN

Command-Recovery Information 806: An area that includes command-specificrecovery information used for command retry.

Previous Response Code (PRC) 808: The resulting response code for thelast command to execute with the specified retry index. The objectcontains zeros when the retry index is assigned, but not in use.

Previous Status Conditions (PSC) 810: The resulting status conditionsfor the last command to execute with the specified retry index when theprevious response code is a preferred value (e.g., 254 or 255).Otherwise, the object contains zeros.

Previous Duplexing Deactivated Indicator (PDDI) 812: The resultingduplexing deactivated indicator for the last command to execute with thespecified retry index. The object is zero, when the retry index isassigned, but not in use.

Retry Version Number (RVN) 816: A value provided by the program andstored in the retry buffer as part of the retry data. The retry versionnumber indicates command-execution status. When the retry version numbermatches the value in the message-command block, the command hascompleted, and the completion status is indicated by the values of thePRC and PSC fields. The retry version number is initialized to the valueX′00′, in one example, when the retry buffer is placed in theassigned-but-not-in-use state.

Retry-Buffer Authority (RBAU): A value set by the program when a retrybuffer is assigned. The initial value of the retry-buffer authority iszero. In this example, RBAU is a separate component of the Retry-Buffer.

Cache Structure

In addition to global objects and retry objects, various objects areassociated with individual types of structures of the coupling facility,such as, for instance, cache and list structures. Various of theseobjects are described below. Again, more, less or different objects mayexist.

Examples of cache structure objects, located within a cache structure,that are associated with duplexing include, for instance:

Duplexing State (DPLXST): A value that indicates the duplexing state forthe cache structure. It has the following encoding:

0 The cache structure is in the simplex state; 1 The cache structure isin the duplexing-active state.

The duplexing state is set to the simplex state or the duplexing-activestate in correspondence with the duplexing-vector bit located at theoffset in the duplexing vector equal to the value of the structureidentifier (SID). The structure is in the simplex state, when theduplexing-vector bit is zero, and the structure is in theduplexing-active state, when the duplexing-vector bit is one.

Remote-Facility Node Descriptor (RFND): A value that includes thenode-descriptor object of the remote coupling facility that has theduplexed copy of the cache structure. The remote-facility nodedescriptor is set, when duplexing is activated, and may be updated,while duplexing is active by an Activate-Duplexing command (describedhereinafter). The initial state is zero.

Remote-Facility Structure Authority (RFSAU): A value that includes thestructure authority of the duplexed copy of the cache structure. Theremote-facility structure authority is set, when duplexing is activated,and may be updated, while duplexing is active by the Activate-Duplexingcommand. The initial state is zero.

Remote-Facility Structure Identifier (RFSID): A value that includes thestructure identifier of the duplexed copy of the cache structure. Theremote-facility structure-identifier is set, when duplexing isactivated, and may be updated, while duplexing is active by theActivate-Duplexing command. The initial state is zero.

Remote-Facility System Identifier (RFSYID): A value that includes thesystem identifier of the remote coupling facility that has the duplexedcopy of the cache structure. The remote-facility system-identifier isset, when duplexing is activated and may be updated, while duplexing isactive by the Activate-Duplexing command. The initial state is zero.

List Structure

Examples of list structure objects, located within a list structure,that are associated with duplexing include, for instance:

Duplexing State (DPLXST): A value the indicates the duplexing state forthe list structure. It has the following encoding:

0 The list structure is in the simplex state; 1 The list structure is inthe duplexing-active state.

The duplexing state is set to the simplex state or the duplexing-activestate in correspondence with the duplexing-vector bit located at theoffset in the duplexing vector equal to the value of the structureidentifier (SID). The structure is in the simplex state, when theduplexing-vector bit is zero, and the structure is in theduplexing-active state, when the duplexing-vector bit is one.

Remote-Facility Node Descriptor (RFND): A value that includes thenode-descriptor object of the remote coupling facility that has theduplexed copy of the list structure. The remote-facility node descriptoris set, when duplexing is activated, and may be updated, while duplexingis active by the Activate-Duplexing command. The initial state is zero.

Remote-Facility Structure Authority (RFSAU): A value that includes thestructure authority of the duplexed copy of the list structure. Theremote-facility structure authority is set, when duplexing is activated,and may be updated, while duplexing is active by the Activate-Duplexingcommand. The initial state is zero.

Remote-Facility Structure Identifier (RFSID): A value that includes thestructure identifier of the duplexed copy of the list structure. Theremote-facility structure-identifier is set, when duplexing isactivated, and may be updated, while duplexing is active by theActivate-Duplexing command. The initial state is zero.

Remote-Facility System Identifier (RFSYID): A value that includes thesystem identifier of the remote coupling facility that has the duplexedcopy of the list structure. The remote-facility system-identifier isset, when duplexing is activated, and may be updated, while duplexing isactive by the Activate-Duplexing command. The initial state is zero.

Message-Path Commands

Messages are communicated between a system, such as system 106 (FIG. 1),and one or more coupling facilities via a message command/responseblock. In one example, a message command/response block has a messagecommand block, which includes a command block and a plurality of requestoperands; a message response block, which includes a response descriptorand a plurality of response operands; and an optional data block.

The message command/response blocks are employed by commands used formessaging. Examples of such commands include an Activate Message Path(AMP) command and an Identify Message Path (IMP) command, which aredescribed below. The commands are issued by an operating system andexecuted by a coupling facility.

Activate Message Path (AMP)

One example of the request operands provided in the message commandblock for the AMP command are summarized in the following table.

Request Operands Acronym Message Header Command Code CC Source NDControl SRCNDC Node Descriptor ND System Identifier SYID Message-pathIdentifier MPID Source Node Descriptor SRCND

In the above table, a source-node-descriptor control is listed. Sincethis operand has been added for duplexing, it is described below.

Source-Node-Descriptor Control (SRCNDC): A value that controls thesetting of the contents of the source-node-descriptor object in themessage-path status vector for the message path designated by theActivate-Message-Path command. It has the following encoding:

0 No source-node-descriptor is provided. The state of the SRCND objectis empty; 1 The source-node-descriptor operand is placed in the SRCNDobject.

In execution of one embodiment of the Activate Message Path command, thenode-descriptor operand is compared to the node-descriptor object, andthe designated message path is compared to the message path used forcommunication. If both are the same, the message path enters the activestate, the system identifier is placed in the message-path statusvector; the source node descriptor is placed in the SRCND object, whenthe SRCNDC operand is B′1′; the SRCND object is placed in the emptystate, when the SRCNDC operand is B′0′; the message-path request levelis placed in the MPRL operand; and response code 0 is stored in theresponse-code operand. Otherwise, the message-path state is not changed,and response code 1 is stored.

The following response codes may be returned:

Message path activated—MPRL is returned;

Node Descriptor or MPID mismatch.

Note for the AMP Command:

-   -   1. When the Activate-Message-Path command is issued by the CFCC        to a remote coupling facility, the SYID request operand is set        equal to the value of the system-identifier sub-object of the        authority object.        Identify Message Path (IMP)

One example of the request operands provided in the message commandblock for the IMP command are summarized in the following table anddescribed herein.

Request Operands Acronym Message Header Command Code CC Node DescriptorMessage-Path Identifier Message-Path State Message-Path Request Level

In execution of one embodiment of the Identify Message Path, the nodedescriptor for the coupling facility is stored in the ND operand; theidentifier of the message path used for communication is stored in theMPID operand; the message-path state is placed in the MPS operand; themessage-path request level is stored in the MPRL operand; the systemidentifier is moved from the authority object to the SYID operand; thesignaling-vector token is stored in the SVT operand; the contents areplaced in the SRCND operand, when the source-node-descriptor object isnot empty; and zeros are stored in the SRCND operand, when thesource-node-descriptor object is empty.

The following response codes may be returned:

Message path information returned.

The response operands provided in the message response block for the IMPcommand are summarized in the following table.

Response Operands Acronym Response Descriptor Response Code RCMessage-Path State MPS Message-Path Request Level MPRL Message-PathIdentifier MPID Node Descriptor ND System Identifier SYIDSignaling-Vector Token SVT Source Node Descriptor SRCNDGlobal Commands for Duplexing

In addition to the messaging commands described above, there are othertypes of commands that are employed for processing associated withcoupling facilities. One type of commands is global commands forduplexing, which are used to control the duplexing of coupling facilitystructures (e.g., activate duplexing, deactivate duplexing, etc.). Thesecommands are described below, but before describing the commands,various global operands are described, which are employed by theduplexing commands, as well as other coupling facility commands.

Global Operands

Examples of global operands used as request/response operands forcommands associated with coupling facility processing are describedbelow:

Comparative Authority (CAU): A value that is compared with the value ofthe authority control.

Comparative Structure Authority (CSAU): A value that is compared withthe value of the structure-authority control.

Comparative Remote-Facility Structure Authority (CRFSAU): A value thatis compared with the value of the remote-facility-structure-authoritycontrol.

Connection Operation Request Type (COPRT): A value that determines thetype of connection operation that is requested on aProbe-Remote-Facility-Connection (PRFC) command (described hereinafter).It has the following encoding:

0 Verify remote-facility attachment; 1 Drop remote-facility connection.

Data-Block Size (DBS): A value that specifies the size of the data blockas an integral multiple of, for instance, 4096-byte units. Valid valuesrange from 1 to 16, in this example.

Preserve-Duplexing Indicator (PDI): A value that indicates whether theduplexing vector and the signaling vector should be preserved, when aSet-Facility-Authority command updates the authority control. It has thefollowing encoding:

0 Reset both the duplexing vector and the signaling vector; 1 No changeis made to the duplexing vector or the signaling vector.

Read CFIB Type (RCFIBT): A value that determines the type ofcoupling-facility information block returned by aRead-Connected-Facility-Controls (RCFC) command (described hereinafter).It has the following encoding:

0 Return remote facility controls only; 1 Return remote facilitycontrols and signal counters.

Remote-Facility Node Descriptor (RFND): A value that includes thenode-descriptor object of the remote coupling facility that has theduplexed copy of the structure. The remote-facility node descriptor isto be different from the local node-descriptor object.

Remote-Facility Structure Authority (RFSAU): A value that includes thestructure authority of the duplexed copy of the structure.

Remote-Facility Structure Identifier (RFSID): A value that includes thestructure identifier of the duplexed copy of the structure.

Remote-Facility System Identifier (RFSYID): A value that includes thesystem identifier of the remote coupling facility that has the duplexedcopy of the structure.

Examples of duplexing commands include an Activate Duplexing Command, aDeactivate Duplexing Command, a Probe Remote Facility Connectioncommand, a Read Connected Facility Controls command, a Read DuplexingVector command, a Test Remote Facility Access command, and a SetRetry-Buffer Authority command, each of which is issued by an operatingsystem and executed by a coupling facility and described below.

Activate Duplexing (ADPLX)

One embodiment of various request operands provided in the messagecommand block for the ADPLX command are summarized in the followingtable and described herein.

Request Operands Acronym Message Header Command Code CC StructureIdentifier SID Remote-Facility Structure Identifier RFSIDRemote-Facility System Identifier RFSYID Remote-Facility Node DescriptorRFND Comparative Structure Authority CSAU Remote-Facility StructureAuthority RFSAU

Moreover, one embodiment of the logic associated with the ADPLX commandis described below with reference to FIG. 9.

Initially, a determination is made as to whether the structure typepermits duplexing, INQUIRY 900. For instance, if the structure is a liststructure that includes a list set, and a Programmable List EntryIdentifier (PLEIDI) control is B′0′, the structure type does not permitduplexing to be activated. In this case, the command is completedwithout updating any objects and a response code (e.g., 3) is returned,STEP 902.

When the structure type permits duplexing to be activated, the structureauthority is compared with the CSAU operand, STEP 904. If the comparisonsucceeds, INQUIRY 906, the remote-facility node descriptor, theremote-facility structure authority, the remote-facility structureidentifier, and the remote-facility system identifier request operandsare placed in the structure controls, STEP 908, and the accessibility ofthe remote facility is determined, INQUIRY 910.

If the structure-authority comparison succeeds and the designated remotefacility is connected, the duplexing-active bit in the duplexing vectoris set to one, STEP 912, and a response code (e.g., 0) is returned, STEP902.

If the structure-authority comparison succeeds, but the designatedremote facility is not connected, INQUIRY 910, then the duplexing-activebit in the duplexing vector is not updated, and a response code(e.g., 1) is returned, STEP 902.

If the structure authority is not equal to the CSAU operand, INQUIRY906, then the command is completed without updating any objects. Thevalue of the structure authority and a response code (e.g., 2) arereturned, STEP 902.

The following response codes may be returned:

Remote facility is connected

Remote facility is not connected

CSAU mismatch—Structure authority is returned

Invalid structure type.

Disallowing of Duplexing for List Structures with LEID-Based Addressing

The extension of the command atomicity rules to a duplexed command pairis accomplished by the implementation of latches on structure objects orinternal control structures. Latches are obtained and held in bothstructures until the ready-to-complete signal is exchanged. Reception ofa ready-to-complete signal confirms that latches have been obtained inthe remote facility. However, if the structure is a list structure thathas a list set, but where LEIDs are internally generated (PLEIDI=B′0′),then it may be possible to violate the command atomicity rules, as thefollowing example demonstrates.

Suppose that a duplexed pair of list structures located on couplingfacilities CF0 and CF1 with internally generated LEIDs concurrentlyprocess a Write List Entry (WLE) command pair with an unconditionalcreate request and a Delete List Entry (DLE) command pair with a requestto locate by LEID. Further, suppose that the DLE command specifies anLEID value of X that is identical to the internally generated LEID forthe WLE command. Finally, assume the commands are processed out of orderby the two facilities, where the WLE command is processed first by CF0and the DLE command is processed first by CF1.

The WLE command succeeds in processing the create request, generates alist entry with LEID=X, issues a ready-to-complete signal and maintainsthe latch on the list entry until a read-to-complete signal is receivedfrom the duplexed WLE command. The DLE command is delayed by the latchin CF0.

The DLE command executes in CF1 and does not find a list entry with thespecified LEID. The DLE command completes with a RC=8 condition, issuesa ready-to-complete signal, and waits until a ready-to-complete signalis received from the duplexed DLE command. No latches are held and theWLE command is free to execute.

The WLE command on CF1 succeeds in processing the create request,generates a list entry with LEID=X, and issues a ready-to-completesignal. Both WLE commands recognize the reception of a ready-to-completesignal and both complete with RC=0 and the latches on the list entry arereleased in both structures.

The DLE command in CF0 is now free to execute, finds the newly createdlist entry with LEID=X, deletes the entry and sends a ready-to-completesignal. Both DLE commands recognize the reception of a ready-to-completesignal and complete processing. However, the DLE command from CF0returns a RC=0 condition and the DLE command from CF1 returns an RC=8condition, entry not found. More importantly, the list structures are nolonger synchronized; a list entry with LEID=X exists in CF1, but not inCF0.

The failure to maintain command atomicity across the structures resultsfrom two events:

-   -   1. The lack of an object to latch by the DLE command that        executes in CF1,    -   2. The specification of the ‘next LEID’ to be generated by the        DLE command.

This problem is avoided if PLEIDs are used; the PLEID collision list inthe list structure controls provides the required object forserialization.

One solution to this problem is to disallow duplexing to be activatedfor structures of this type. In this case, the Activate-Duplexingcommand ends with a response code (e.g., 3) condition. For liststructures with list sets, assignment of LEIDs are to be under thecontrol of the operating system and the PLEIDI control is to be b′1′ inthe list structure type.

Deactivate Duplexing (DDPLX)

One embodiment of the request operands provided in the message commandblock for a DDPLX command are summarized in the following table anddescribed herein.

Request Operands Acronym Message Header Command Code CC StructureIdentifier SID Comparative Structure CSAU Authority ComparativeRemote-Facility CRFSAU Structure Authority

Moreover, one embodiment of the logic associated with the DDPLX isdescribed with reference to FIG. 10.

Initially, the CSAU operand is compared with the structure-authorityobject and the CRFSAU operand is compared with the remote-facilitystructure authority object, STEP 1000. If both comparisons aresuccessful, INQUIRY 1002, the duplexing-active bit specified by the SIDoperand is set to zero, STEP 1004, and a response code (e.g., 0) isreturned, STEP 1006. The remote-facility-node-descriptor, theremote-facility-system-identifier, theremote-facility-structure-identifier, and theremote-facility-structure-authority objects are not updated.

If the structure authority is not equal to the CSAU operand or if theremote-facility structure authority is not equal to the CRFSAU operand,INQUIRY 1002, then the structure authority and remote-facility structureauthority are placed in the MRB, STEP 1008. The MRB and a response code(e.g., 2) are returned, STEP 1006.

The following response codes may be returned:

Duplexing deactivated;

CSAU or CRFSAU mismatch—Structure authorities are returned.

Probe Remote Facility Connection (PRFC)

One embodiment of the request operands provided in the message commandblock for the PRFC command are summarized in the following table anddescribed herein.

Request Operands Acronym Message Header Command Code CC ConnectionOperation Request Type COPRT Retry Index RX Retry Version Number RVNRemote-Facility Node Descriptor RFND Remote-Facility System IdentifierRFSYID

Further, one embodiment of the logic associated with the PRFC command isdescribed with reference to FIG. 11.

An operation is performed for the specified remote-facility connectionand the results of the operation are returned in the response block.

The operation that is performed depends on the connection operationrequest type. When the COPRT operand is zero, INQUIRY 1100, theremote-facility attachment is verified, STEP 1102, as follows:

-   -   1. A message path in the remote facility path group is selected.    -   2. An Identify Message Path (IMP) command (described herein) is        issued to the remote facility on the selected message path.    -   3. If the IMP command completes successfully, the ND, SYID and        SVT response operands are compared with the RFND, RFSYID, and        RFSVT remote facility controls. If the controls match, the        remote-facility is verified as attached and the command        completes. A response code (e.g., 0) is returned.    -   4. If the IMP command fails to complete successfully, or if the        values returned do not match the corresponding remote-facility        controls, another path in the path group is selected and another        IMP command is issued.    -   5. If no IMP command has successfully verified the attachment,        and if no paths remain in the path group, the attachment        verification fails. The command completes and a response code        (e.g., 1) is returned.

When the COPRT operand is one, INQUIRY 1100, the remote-facilityconnection is dropped, STEP 1104, by initiating a coupling facility linkinitialization operation for each coupling facility link with anavailable sender CHPID at the local coupling facility and an availablereceiver CHPID at the remote coupling facility. (In one example, thisstep is performed by a channel subsystem.) Model-dependent means areused to initiate the coupling facility link initialization procedure.When the coupling facility link initializations have been successfullyinitiated, the command completes and a response code (e.g., 0) isreturned. After the link initialization operations have completed, theremote-facility connection may be reestablished at any time.

When the retry index is non-zero, INQUIRY 1106, the retry buffer iswritten, STEP 1108.

The following response codes may be returned, STEP 1110:

-   -   Operation Successful—Retry version number returned;    -   Attachment not verified—Retry version number returned.        Note for the PRFC Command:    -   1. The Probe-Remote-Facility Connection operation is performed        as a single continuous process. In some cases, the length of the        process may exceed a 300 millisecond time limit and an interface        control check may be presented before the operation completes.        The retry buffer is used in this case to present the results of        the operation to the program. Command-quiescing rules ensure        that the retry buffer cannot be read before the process is        completed and the MRB is stored in the retry buffer.        Read Connected Facility Controls (RCFC)

One embodiment of various request operands provided in the messagecommand block for the RCFC command are summarized in the following tableand described herein.

Request Operands Acronym Message Header Command Code CC Read CFIB TypeRCFIBT Data Block Size DBS

In execution of one embodiment of the RCFC command, when sufficientmessage-buffer space is provided, a connected-facility information block(CFIB) is added to the data block and the processed count is increasedby one for each connected coupling facility.

When the RCFIBT operand is, for instance B′0′, a 128-byte CFIB includingthe remote facility controls is stored in the data block. One example ofa CFIB, when the RCFIBT is B′0′ is, as follows:

Data Operands Acronym Remote-Facility System Identifier RFSYIDRemote-Facility Signaling Vector Token RFSVT Remote-Facility Path GroupSize RFPGS Remote-Facility Controls Time of Creation RFCTOCRemote-Facility Path Descriptor #1 RFPD 1 Remote-Facility PathDescriptor #2 RFPD 2 . . . . . . Remote-Facility Path Descriptor # PGSRFPD PGS Remote-Facility Node Descriptor RFND

When the RCFIBT operand is, for instance, B′1′, a 256-byte CFIBincluding the remote-facility controls and signal counters is stored inthe data block. One example of the CFIB, when the RCFIBT is B′1′ is, asfollows:

Data Operands Acronym Remote-Facility System Identifier RFSYIDRemote-Facility Signaling Vector Token RFSVT Remote-Facility Path GroupSize RFPGS Remote-Facility Sender Path Group Size RFSPGS Remote-FacilityControls Time of Creation RFCTOC Remote-Facility Path Descriptor #1 RFPD1 Remote-Facility Path Descriptor #2 RFPD 2 . . . . . . Remote-FacilityPath Descriptor # PGS RFPD PGS Remote-Facility Node Descriptor RFNDReady-to-Execute Signal Counter RTESC Ready-to-Complete Signal CounterRTCSC Halt-Execution Signal Counter HESC Request-for-Suppression SignalCounter RFSSC Request-for-Suppression Accepted Signal RFSASC CounterSignal-Service Time First Moment SSTFM Signal Service Time Second MomentSSTSM Delayed Signal Counter DSC Signal Delay Time First Moment SDTFMSignal Delay Time Second Moment SDTSM Signal Redrives Signal CounterSRDSC Remote-Facility Channel Path Identifier #1 RFCHP 1 Remote-FacilityChannel Path Identifier #2 RFCHP 2 . . . . . . Remote-Facility ChannelPath Identifier # PGS RFCHP PGS Remote-Facility Sender Path Descriptor#1 RFSPD 1 Remote-Facility Sender Path Descriptor #2 RFSCHP 2 . . . . .. Remote-Facility Sender Path Descriptor # SPGS RFSPD SPGSRemote-Facility Sender Channel Path Identifier RFSCHP 1 #1Remote-Facility Sender Channel Path Identifier RFSCHP 2 #2 . . . . . .Remote-Facility Sender Channel Path Identifier RFSCHP SPGS # SPGS

When all connected coupling facility controls have been placed in a CFIBlist, the CFIB list, the processed count and a response code (e.g., 0)are returned to the program.

When the data block is full and additional connected facility controlsare to be added to the list, the CFIB list, the processed count and aresponse code (e.g., 4) are returned to the program.

When the product of the value of the DBS operand and 4096 is larger thanthe message-buffer size, there is insufficient message-buffer space tocontain the data block. In this case, the command is completed and aresponse code (e.g., 11) is returned.

The following response codes may be returned:

-   -   All CFIBs returned—CFIB list and processed count are returned;    -   Data-block size too small—CFIB list and processed count are        returned;    -   Insufficient message-buffer space.        Read Duplexing Vector (RDV)

The request operands provided in the message command block for the RDVcommand are summarized in the following table and described herein.

Request Operands Acronym Message Header Command Code CC Data Block SizeDBS

In execution of one embodiment of the RDV command, when sufficientmessage-buffer space is provided, the duplexing vector is placed in thedata block, the SID limit is placed in the Sid Limit (SL) operand, and aresponse code (e.g., 0) is placed in the Response Code (RC) operand.

When the product of the value of the DBS operand and 4096 is larger thanthe message-buffer size, there is insufficient message-buffer space tocontain the data block. In this case, the command is completed and aresponse code (e.g., 11) is returned.

The following response codes may be returned:

-   -   Duplexing vector is returned;    -   Insufficient message-buffer space.        Test Remote Facility Access (TRFA)

The request operands provided in the message command block for the TRFAcommand are summarized in the following table and described herein.

Request Operands Acronym Message Header Command Code CC Remote-FacilitySystem Identifier RFSYID Remote-Facility Node Descriptor RFND

In execution of one embodiment of the TRFA command, when the remotefacility accessibility level is connected, a response code (e.g., 0) isreturned. When the remote facility is not connected, the remote-facilitydisconnect time, the remote-facility-disconnect-time validity indicator,and a response code (e.g., 1) are returned.

The following response codes may be returned:

-   -   Remote Facility Connected;    -   Remote Facility Not Connected—Remote-Facility Disconnect Time        returned.        Set Retry-Buffer Authority (SRBA)

The request operands provided in the message command block for the SRBAcommand are summarized in the following table and described herein.

Request Operands Acronym Message Header Command Code CC Retry Index RXComparative Retry-Buffer Authority CRBAU Retry-Buffer Authority RBAU

One embodiment of the logic associated with the SRBA command isdescribed with reference to FIG. 12.

Initially, a determination is made as to whether the RBAU control, andCRBAU and RBAU operands are zero, INQUIRY 1200. If not, then processingcontinues with a comparison of the retry-buffer-authority control valueto the CRBAU operand, STEP 1201. When they compare as equal, INQUIRY1202, the value of the RBAU operand is stored in the retry-bufferauthority, the signaling-vector entry associated with the retry index isreset, and the retry-buffer state is updated, STEP 1204.

When the retry-buffer-authority control is changed from zero to anon-zero value, INQUIRY 1206, the retry-buffer state is changed fromunassigned to assigned and not in use, STEP 1208, the bit in the retryvector addressed by the retry index is set to one, STEP 1210, and aresponse code (e.g., 0) is returned, STEP 1212.

When the retry-buffer-authority control is changed from a non-zero valueto zero, the retry-buffer state is changed to the unassigned state, STEP1214, the bit in the retry vector addressed by the retry index is set tozero, STEP 1210, and a response code (e.g., 1) is returned, STEP 1212.

When the retry-buffer-authority control is changed from a non-zero valueto a non-zero value, the retry-buffer state is set to assigned and notin use, STEP 1216, and a response code (e.g., 2) is returned, STEP 1212.

When the compare fails, INQUIRY 1202, the retry-buffer state is notchanged. The value of the retry-buffer authority is placed in the RBAUoperand, STEP 1218. The RBAU operand and a response code (e.g., 4) arereturned, STEP 1212.

When the retry-buffer-authority control is initially zero and both theCRBAU and RBAU operands are zero, INQUIRY 1200, the retry-buffer remainsin the unassigned state, and a response code (e.g., 3) is returned, STEP1212.

The following response codes may be returned:

-   -   Retry buffer enters assigned state;    -   Retry buffer enters unassigned state;    -   Retry buffer remains assigned;    -   Retry buffer remains unassigned;    -   Comparison failed—Retry-Buffer Authority returned.        Notes on Set Retry Buffer Authority Command:    -   1. When a cache or locking command completes with an interface        control check, the MRB is lost, and thus the value of the        current-signal-group index cannot be determined. The operating        system can reset the signaling-vector entry by issuing a        Set-Retry-Buffer-Authority command, specifying the retry index        that was used by the command and setting the CRBAU and RBAU        operands equal to the RBAU object.    -   2. When a command completes with interface control check and the        command was a duplexed command, the signaling-vector entry in        the coupling facility associated with the duplexing retry index        and the duplexing signal group index is in an unknown state.        Signals may continue to be sent from the coupling facility that        presented the interface control check until the command clear        status is recognized. The operating system can reset the        signaling-vector entry by issuing a Set-Retry-Buffer-Authority        command. However, the execution of the SRBA command is to be        delayed until at least one command is observed to complete at        the coupling facility that presented the IFCC. An example        sequence of this recovery is the following:        -   i. A duplexed command is sent to CF 1 and CF 2.        -   ii. The command on CF 1 completes, but the command on CF 2            ends with an IFCC.        -   iii. An IMP command is issued to CF 2 and completes. The            rules for command quiesce in the coupling facility require,            in one example, the duplexed command to be completed or            suppressed, prior to executing the IMP command. So,            completion of the IMP command indicates that the duplexed            command is no longer executing in CF 2.        -   iv. The SRBA command is issued to CF 1 to reset the            signaling vector entry associated with the retry index and            current signal-group index in CF 1.        -   v. The SRBA command is issued to CF 2 to reset the signaling            vector entry associated with the retry index and current            signal-group index in CF 2. The two SRBA commands can be            issued in parallel.        -    An alternate sequence would be for the IMP command sent to            CF 2 to be replaced by the SRBA command. This would            eliminate one step, but would place an ordering requirement            on the retry-buffer commands sent to each coupling facility.    -   3. If the IMP command issued in step iii in the previous note        fails to complete, then connectivity to the structure in CF 2 is        lost and duplexing is broken. However, one is to guarantee that        no signals are being sent from CF 2 to CF 1 before the SRBA        command can be issued to CF 1. This can be accomplished by        issuing a Probe-Remote-Facility-Connection command to CF 1,        specifying CF 2 as the remote facility. This can be done in two        steps, with the first step requesting the operation of verifying        the attachment. This operation is benign in that only IMP        commands are issued and no state changes are made to any        objects. However, this command may not succeed, in which case        the operation of dropping the connection should be invoked. This        operation is guaranteed to complete.

FIGS. 13 a & 13 b describe one embodiment of the flow of control in theOS for handling interface control checks (IFCC). First, the OS waits forboth operations to complete at the subchannel, STEP 1300. If both endwith normal completions, INQUIRY 1302, as indicated by, for instance,valid duplexing signal group indices (DSGXs) (described below), then theduplexing signal group index that is to be used by the next operationselecting these subchannels (and associated retry index) is updated ininternal controls to the values returned in the CSGX fields in therespective response descriptors, STEP 1304. At that point, thesubchannels are made available for use by the next operation. However,if one or both commands complete with an interface control check, thenthe signaling vector entries associated with the subchannel and retryindex in each facility are recovered before the subchannels are madeavailable.

One embodiment of the recovery flow is described next. If the primaryoperation ended with an IFCC, INQUIRY 1306, then the connection to theprimary coupling facility is tested by issuing an IMP command, STEP1308. Successful completion of the IMP command implies two things.First, the connection itself is verified. Second, the command quiescingrules dictate that execution of the IMP command indicates that thecommand being recovered has completed. So, in particular, no signals arebeing issued by CF 1 on behalf of that command.

Should the primary coupling facility be connected or if the primaryoperation did not end with an IFCC, then a similar check is made of thesecondary command, INQUIRY 1310. If the secondary command ended with anIFCC, then the connection to the secondary coupling facility is tested,INQUIRY 1312. If the secondary command succeeded successfully, or if thesecondary command had an IFCC and the subsequent IMP command succeeded,then all signals are guaranteed to have stopped flowing and the signalgroup index in each signaling vector can be reset by issuing an SRBAcommand to each coupling facility, STEP 1314. If, however, the secondarycommand had an IFCC and the IMP command fails, then aProbe-Remote-Facility command (PRFC) is issued to the primary couplingfacility, STEP 1316. This is described further with reference to FIG. 13b.

Initially, the PRFC command is issued with the verify option, STEP 1350.If unsuccessful, INQUIRY 1352, the OS escalates to the function ofdropping the remote facility connection, STEP 1354. In one example, thisis done by redriving the PRFC. Forcing this connection to be droppedensures that the remote coupling facility recognizes a state change inthe peer ISC link which stops any signals from being issued once theconnection is dropped. (It should be noted that the connection can beimmediately reestablished, so long as the state transition is made inthe connection.)

Subsequent to redriving the PRFC or if the initial PRFC is successful,then the SRBA command is issued to reset the primary DSGX value, STEP1318 (FIG. 13 a).

Returning to INQUIRY 1308, if there is no connection to the primarycoupling facility, then there is a further determination is made as towhether there is a connection to the secondary coupling facility,INQUIRY 1360. If there is no connection to either facility, then nofurther action is required, STEP 1361.

However, if there is connectivity to the secondary facility, then aProbe-Remote-Facility command is issued to the secondary couplingfacility, STEP 1362, as described further with reference to FIG. 13 b.

Similar to the logic described above, the PRFC command is issued withthe verify option, STEP 1364. If the PRFC command is unsuccessful,INQUIRY 1366, then the operating system escalates to the function ofdropping the remote facility connection, STEP 1370. Thereafter, or ifthe initial PRFC is successful, then an SRBA command is driven to resetthe secondary DSGX value, STEP 1368 (FIG. 13 a).

Once the SRBA command has been successfully completed to each couplingfacility that had an IFCC, the DSGX value associated with the subchannelis set to one and the subchannels are made available for reuse. In thiscase, they will most likely be reused for retrying the failed duplexedoperation. But, other subchannels may also be selected.

Duplexing Signal Processing

Duplex-command pairs, when they execute in the coupling facilities,exchange duplex signals between the coupling facilities. In oneembodiment, the list notification command is used to exchange the duplexsignals. Further, to facilitate exchanges of duplex signals, eachcoupling facility has one or more sender channels and one or morereceiver channels that connect the two coupling facilities. (See, e.g.,FIG. 4.) The duplex signals are used to extend the object and commandconcurrency rules across the synchronized objects in each couplingfacility. The set of duplex signals and their protocols are describednext.

Duplex Signaling

Signaling Vector Entries

In one embodiment, the duplex signaling protocol employs a signalingvector, which has a plurality of entries. In one example, there is asignaling vector entry for each retry buffer.

The duplex commands use signal groups in the signaling vector tocoordinate the progress of the first and subsequent entries in acommand. In a duplex command, there is a signal-group-index value whichindicates which of the signal groups is the current-signal group in theduplex coupling facility. The current-signal-group-index valuedesignates the current signal group in the recipient coupling facility.The value of this index points to the signal group that is to be usedfor the entry of the command in the respective coupling facility'ssignaling vector. The signal-group indexes are incremented as part ofthe process of committing an entry, or completing the command. When thesignal-group-index object reaches three and increments, the value wrapsto one.

The Set-Retry-Buffer-Authority command (described above) is used to setthe signal-group values for all signal-group indexes to zero and resetthe signal-group index to one.

Duplex commands return the incremented signal-group-index value in theMRB. This value is used in subsequent duplex commands as thecurrent-signal-group-index value.

In one embodiment, the LFSS component manages the assignments ofsignal-group-index values.

Notes on Signaling Vector Entries:

-   -   1. Retry indexes are assigned on a subchannel basis. A single        retry index value is not used by two different operating system        images.    -   2. The LFSS component, as one example, maintains a table of        index values to be used for the next issuance of a command on a        subchannel. This means that the operating system remembers one        signal group index value per subchannel (or per RX,        equivalently).        Duplex Signaling Operands

This section defines the signaling operands. Request operand validity ischecked by the receiving coupling facility. Detection of invalid valuesresults in the command failing.

One embodiment of duplex signaling operands, used in accordance with anaspect of the invention, include, for instance:

Duplex Retry Index (DRX): A value that designates the duplex retrybuffer. The duplex retry index is provided as a request operand on theduplexing command.

Duplex Signal Group Index (DSGX): A value that identifies the duplexsignal group to be updated in the signaling vector. Valid DSGXs areassigned DSGX values within the range of one to three, as an example. Aninvalid DXGX does not update the signal-vector.

Duplex Signal (DS): A value that indicates the duplex signal to be setin the signaling vector. Valid values are, for instance, one throughfive. An invalid value reults in a signal vector bit not being set. Ithas the following encoding, as example:

000 Invalid 001 Ready to execute 010 Ready to complete 011 Request forsuppression 100 Request for suppression accepted 101 Halt execution110-111 InvalidDuplex Signals

Duplex signals coordinate the execution of duplex commands in each ofthe two coupling facilities. In one embodiment, the halt-executionsignal has precedence over the other duplex signals.

Duplex signals are communicated, for instance, by using listnotification commands, which are generated secondary message commands.For example, the duplex signals are sent using the list notificationcommand with a nonempty-state-change operand value of zero, asummary-update operand value of zero, the signal-vector token for theremote coupling facility, and a list-notification-entry-number operand.One embodiment of a format of the duplex signal command block of alist-notification command is shown in FIG. 14.

A successful MRB returned for a duplex signal indicates that the signalwas received in the signal vector. However, it does not indicate, inthis example, that the signal was recognized by the duplex command atthe duplex coupling facility.

Signals are used by the duplex-command pair to communicate the progressof the command in the respective coupling facility.

Association of Systems, Retry Buffers, and Signal Groups

FIG. 15 depicts one example of an association of systems to retrybuffers, signaling-vector entries and signal groups. Each of the retrybuffers in a coupling facility is assigned to be available to oneoperating system image. This is accomplished by assigning a range ofretry buffer indices (RX) to each system at the point in time when thesystem joins the Parallel Sysplex. Retry buffers are assigned by thez/OS image on a subchannel basis. For example, each system takes anumber of retry buffers and assigns them internally. One example of anassignment of retry indices is depicted in FIG. 15, in which indexvalues 1 to 5 are assigned to System 1, values 6 to 12 are assigned toSystem 2, values 13 to 19 are assigned to System 3, and values 110 to115 are assigned to System N.

The association of retry buffers to signaling vector entries is aone-to-one correspondence and the partitioning of the retry buffers by asystem defines a corresponding partitioning of the signaling vectorentries. The retry index then becomes the index for both the retrybuffer and the signaling vector entry. For instance, a retry index valueof 14 points to both a retry buffer and a signaling vector entry, asdepicted in FIG. 15.

FIG. 15 also shows the layout of the physical signaling vector. Anextended logical view is shown in FIG. 6. As FIG. 15 depicts, thephysical signaling vector entry is further divided into three signalgroups indexed by the signal group index. Each signal group contains aneight bit value that is set by the signaling commands and checked andcleared by the signaling protocol engine, as described herein.

Duplex Commands

Each of the duplex commands in the duplex-command pair designates aretry buffer and a signal group to coordinate the signaling between thetwo coupling facilities. Each duplex command includes the retry-indexfor the coupling facility that is receiving the command and theretry-index and signal-group index for the coupling facility that hasthe duplexed structure. The current-signal-group index is used for thecoupling facility receiving the command. The retry index andsignal-group-index value may not be the same in both couplingfacilities. The set of retry-index and signal-group-index values aremirrored in the commands of the duplex-command pair.

The commands of a duplex-command pair are associated by the retry-indexvalues in the MCB. The signal-group-index values used to do signalingfor these retry buffers are associated by the signal-group-index valuesin the MCBs.

Notes on Duplexing Commands

-   -   1. The retry buffer to subchannel binding prevents multiple        systems (e.g., Central Processing Complexes (CPCs)) from trying        to use the same signal-group bits at the same time.        Duplex Command Execution        The execution of duplex commands is divided into three phases,        each of which is executed in a coupling facility: command        decode, command execution, and command completion phases. The        first phase is command decode. In this phase, the command starts        execution and sends a ready-to-execute command to the other        coupling facility. The ready-to-execute signal is sent to        determine if the command has been command suppressed at the        other coupling facility. If no ready-to-execute or        ready-to-complete signal is recognized, the command does not        proceed beyond the command decode phase and the command can be        terminated without breaking duplexing. The ready-to-execute        signal is sent, when, for instance, the following conditions are        met: (1) a path to the duplex-coupling facility is active; (2)        the SID is active, (3) duplexing is active; (4) a path in the        duplex class path group is active; (5) dumping serialization is        not active. The ready-to-complete signal is used as command        decode completion indication due to the fact that the        ready-to-execute and ready-to-complete signals can be sent in        parallel on different channel paths.

The execution phase of the command is different for single entrycommands and list-form commands (i.e., commands that process multipleentries one at a time). For single entry commands other thanWrite-And-Register and Write-When-Registered, after the command decodephase is completed, a single ready-to-complete signal is sent. When theready-to-complete signal has been sent and a ready-to-complete signalhas been recognized, the entry is committed and the command enters thecommand completion phase. For the Write-And-Register andWrite-When-Registered commands, the execution phase is dependent on thesetting of the WRTCI request operand (described below). When the WRTCIrequest operand is B′0′, the execution phase is substantially identicalto the execution phase for any single-entry command. When the WRTCIrequest operand is B′1′, the sending of the ready-to-complete signal isdelayed until both the command decode phase is complete and aready-to-complete signal or a halt-execution signal is recognized. Whena ready-to-complete signal is recognized and the command decode phase iscomplete, a ready-to-complete signal is sent and the command enters thecommand completion phase. When a halt-execution signal is recognized,the command execution is halted.

For list-form commands, after the command decode phase is completed, aready-to-complete signal is sent. When the ready-to-complete signal hasbeen sent and a ready-to-complete signal has been recognized, the entryis committed and another ready-to-complete signal is sent. This exchangeof ready-to-complete signals continues until the last entry is committedor a halt execution signal is recognized, then the command enters thecommand completion phase.

In the command completion phase, the current-signal index is incrementedand the MRB for the command is returned with the appropriate responsecode.

Note on Duplexing Command Execution:

-   -   1. Inconsistency of response codes from the two coupling        facilities for duplexed command pairs are reconciled. In some        instances, no action is taken, and in others, duplexing is        broken. The operating system is responsible for performing this        function.        Duplexing Processes

During the various phases of duplex command execution, one or moreduplexing processes may be invoked. Examples of processes that may beexecuted are described below.

No Command Active Process

Since a duplex-command pair is issued on two separate couplingfacilities and each of the coupling facilities services these commandsat differing rates, it is possible for a coupling facility to receivesignals for a duplex command that has not yet started at the receivingcoupling facility. For example, the ready-to-execute signal can bereceived by a coupling facility that has not yet started commandexecution of a duplex command.

The duplex command when it executes examines the signals in thecurrent-signal group to see if any have been received prior to commandexecution and continues execution based on the received signals for thecommand. The duplex command may also time-out and be invalidated priorto the command being executed at the coupling facility after receivingthese signals.

Entry Commit Process

An entry is committed when the entry object is updated in the couplingfacility and can be observed by subsequent accesses of the entry. Priorto an entry being committed, the entry remains unchanged as observed bysubsequent accesses of the entry.

The current-signal-group index and the duplex-signal-group index areincremented by one, when an entry is committed.

Single Entry Duplex Command Process

When a duplex command executes and it is ready to start processing foran entry, a ready-to-execute signal is sent. The ready-to-execute signalis sent when, for instance, the following conditions are met: (1) a pathto the duplex-coupling facility is active; (2) the SID is active; (3)duplexing is active; (4) a path in the duplex class path group isactive; (5) dumping serialization is not active. The ready-to-executesignal, when it is recognized, indicates that the duplex command at theother coupling facility has started execution of the command and aready-to-complete signal can be sent. A ready-to-complete signal cannotbe sent until a ready-to-execute or ready-to-complete signal has beenrecognized. The ready-to-complete signal may be recognized before theready-to-execute due to the fact that they both may be sent in paralleland the ready-to-complete may arrive and be recognized first.

The ready-to-complete signal is sent when the command is ready to commitcompletion status or a request exception condition exists for thecommand. When the ready-to-complete signal has been sent, the MRB forthe ready-to-complete signal has been recognized, and theready-to-complete signal from the other coupling facility has beenrecognized, the entry is committed, and the command completes or commandexecution stops at the completion of command decode and the requestexception is presented for the command.

If a halt-execution or request-for-suppression signal is received, theentry is not committed and command execution stops at the completion ofcommand decode.

If a command has received a ready-to-execute command, is ready to committhe entry and has sent a ready-to-execute command, but not received anMRB in response, the command may send a ready-to-complete command onanother message path. The command cannot send a halt-execution,request-for-suppression, or request-for-suppression-accepted signaluntil the ready-to-execute and ready-to-complete MRBs have beenreceived. Note that a halt cannot be sent after a ready-to-complete hasbeen sent for single entry commands.

The ready-to-complete signal cannot be sent before the ready-to-executesignal has been sent. The ready-to-complete signal may be sent on aseparate message path in parallel with the ready-to-execute signal.

List-form Duplex Command Process

When a duplex command executes and it is ready to start processing foran entry, a ready-to-execute signal is sent. The ready-to-execute signalis sent when, for instance, the following conditions are met: (1) a pathto the duplex-coupling facility is active; (2) the SID is active; (3)duplexing is active; (4) a path in the duplex class path group isactive; (5) dumping serialization is not active. The ready-to-executesignal, when it is recognized, indicates that the duplex command at theother coupling facility has started execution of the command and aready-to-complete signal can be sent. A ready-to-complete signal cannotbe sent until a ready-to-execute or ready-to-complete signal has beenreceived. The ready-to-complete signal may be recognized before theready-to-execute due to the fact that they both may be sent in paralleland the ready-to-complete may arrive and be recognized first.

The ready-to-complete signal is sent when the command entry is ready tocommit completion status for the entry or a request exception conditionexists for the command. When the ready-to-complete signal has been sentand a ready-to-complete signal has been recognized, the entry iscommitted or command execution stops at the completion of command decodeand the request exception is presented for the command. Subsequententries then send and receive ready-to-complete signal pairs. Theready-to-execute signal is exchanged only once, prior to the firstentry.

If one or more entries have been committed and a halt-execution orrequest-for-suppression signal is received, the entry is not committedand command execution stops at the previous entry and a timeout responsecode is returned. If no entries have been committed and a halt-executionor request-for-suppression signal is received, the entry is notcommitted and command execution stops at the completion of commanddecode and a halt response code is returned.

If a command has recognized a ready-to-execute signal, is ready tocommit the entry and has sent a ready-to-execute signal, but notreceived an MRB in response, the command may send a ready-to-completesignal on another message path. The command cannot send ahalt-execution, request-for-suppression, orrequest-for-suppression-accepted signal, until the ready-to-execute andready-to-complete MRBs have been received.

When the ready-to-complete signal has been sent, the MRB for theready-to-complete signal has been recognized, and the ready-to-completesignal from the other coupling facility has been recognized, the entryis committed, and the command completes or advances to the next entry,if additional entries exist. Note that a halt cannot be sent after aready-to-complete has been sent without first advancing to the nextentry. If a ready-to-complete has been sent for the last entry in thelist, then no halt signal can be sent.

The ready-to-complete signal cannot be sent before the ready-to-executesignal has been sent. The ready-to-complete signal may be sent on aseparate message path in parallel with the ready-to-execute signal.

Request for Suppression Process

When a command that is waiting on resources held by other commandsrecognizes a ready-to-complete signal for itself or for a command oflower priority, the command sequence numbers of the other commandsholding resources desired by this command are examined. For each suchcommand, one of three cases can occur, as examples:

-   -   1. If the command sequence number is less than the command        sequence number of the resource owning command, the waiting        command is of higher priority. The resource owning command sends        a request-for-suppression signal.    -   2. If the command sequence number is greater than the command        sequence number of the resource owning command and the resource        owning command has obtained all desired resources, the resource        owning command has priority and no action occurs.    -   3. If the command sequence number is greater than the command        sequence number of the resource owning command and the resource        owning command cannot obtain all desired resources, the resource        owning command recognizes a ready-to-complete signal for the        lower priority command and issues a request for suppression        signal on its behalf.

If the command-sequence number of the waiting command and the selectedresource owning command are equal, then the system identifier associatedwith each system that sent the command is used as a tie breaker. If thesystem identifier of the waiting command is less than the systemidentifier of the selected resource owning command, then the waitingcommand is of higher priority and case 1 applies. Otherwise, cases 2 and3 apply.

The request-for-suppression signal, when received, causes the receivingcommand to either suppress the command or to allow the command tocontinue, if the resources are available. If the command is suppressed,a request-for-suppression-accepted signal is sent and the command issuppressed. If the command does not need to be suppressed, then theready-to-complete or halt execution signal is sent, as is appropriate.

The request-for-suppression signal can only be sent, in this example,after a ready-to-execute signal and ready-to-complete signal have beensent and their respective MRBs have been recognized.

Request-for-suppression-accepted can only be sent, in this example,after a ready-to-execute signal has been sent and the MRB has beenrecognized.

Note on the Request for Suppression Process:

-   -   1. The request-for-suppression signals are sent based on the        assumption that the commands may be in a deadlock situation.        This may or may not be the case, but over-reaction is preferred        to deadlock situations. A command can be legitimately locked out        due to a resource lock from an earlier command holding the        resource and have later commands that have issued        ready-to-complete signals be suppressed, even if they are not        dependent on the latches held for the original command.

Halting Execution

A halt-execution signal can be sent by a coupling facility once alloutstanding signals have received an MRB. A halt-execution signal can besent or received after any other signal, once the ready-to-executesignal exchange has taken place.

When a halt-execution signal is recognized, command execution stops atthe completion of command decode or at the previous entry for list-formcommands after the first entry has been committed. The receiving commandcompletes any outstanding signals for the entry being processed.

A halt condition can occur due to a resource condition, such as any ofthe following, as examples:

-   -   Model dependent timeout.    -   Cache full condition.    -   List set full condition.    -   List full condition.    -   Local Cache Entry (LCE) not registered.    -   Entry not found for the Invalidate-Name-List command and a        halt-invalidation control is set to B′1′, or for an        Unlock-Castout-Locks command and the skip-nonexisting-entries        control is set to B′0′.    -   A request exception with exception code 31, ‘local-cache        identifier not attached’, for the Unlock-Castout-Locks command        and the detachment emulation control is set to B′1′.    -   Failed version number comparison for cache commands.    -   When a link failure or link timeout is recognized on the link        that received the command and the ready-to-execute signal        exchange has taken place.    -   Event Monitor Control (EMC) object space is full.    -   Dump serialization held.    -   Name assignment suppressed in a write and register command.    -   Shared or exclusive access to coupling facility objects is not        available and contention exists with a command with a lower        command sequence number.    -   As well as for other resource conditions.

Any time one of these conditions occurs, a halt-execution signal issent, if a duplex command is being executed.

When a halt-execution signal is sent and the MRB for the halt-executionsignal is recognized, the command is suppressed and the response codedefined for the halt condition in the command is returned with theseexceptions:

-   -   1. If the halt condition is due to a link failure or link        timeout and the command is a single-list-entry command, a        list-form command where no list items have been processed, or a        list-form command with no retry buffer specified, the MRB is        discarded.    -   2. If the halt condition is due to a link failure or link        timeout and the command is a list-form command with a retry        buffer specified and one or more list items have been processed,        the command completes with a model-dependent timeout.    -   3. If the halt condition is due to shared or exclusive access to        coupling facility objects being unavailable and contention        exists with a command with a lower command sequence number, the        command completes with an implicit suppression-request-accepted        condition. If the command is a single-list-entry command or a        list-form command where no list items have been processed, a        response code (e.g., 19) is returned. If the command is a        list-form command and one or more list items have been        processed, the command completes with a model-dependent timeout.

When a halt signal is recognized, the command is backed out, and if itis a list-form command and at least one list item was executed tocompletion, the entry is backed out and the current object index is setto the entry that was backed out, and then the response code is set. Ifboth a command suppression condition, and halt-execution signal occur,the command suppression response code is returned. If only thehalt-execution signal occurs and the command is either asingle-list-entry command or a list-form command where no list itemshave completed execution, the command execution is halted and a responsecode (e.g., 18) is returned.

Signal Group Processing

The duplex-signal-group-index and duplex-retry-index values are sent aspart of the LNEN operands in the list notification command. Thesignal-vector bit in the signaling vector is set based on the LNENvalue. Each duplex-signal list-notification command sets only a singlebit in the signal group. The MRB for the list notification command isreturned once the bit has been set. The recognition of the MRB for thelist notification command is not used as an indication that the signalhas been recognized at the receiver of the signal, it only indicatesthat the bit has been set.

When any of the following conditions exist, the signal group with anindex value that is one greater than the current-signal-group-indexvalue is set to zero:

A ready-to-complete signal is sent.

A request-for-suppression signal is sent.

A request-for-suppression-accepted signal is sent.

A halt-execution signal is sent or recognized.

For signals sent, the signal group is cleared prior to the signal beingsent. When the current-signal-group-index value reaches three and anincremented value is needed, the index value wraps to a value of one.

The duplex commands have a retry index, duplex-retry index, andduplex-signal-group-index value as operands. The retry index andcurrent-signal-group-index values are used to determine the signalingvector entry and signal group in the signaling vector that are to beused in the coupling facility. The current-signal-group index indicateswhich set of signaling bits is the initial set of duplex signals to beused for the command. The duplex-retry index and duplex-signal-groupindex are used as part of the LNEN operand in the list notificationcommand to be sent to the duplex coupling facility.

Notes on Signal Group Processing:

The following signal groups can be actively used in a single command:

The current-signal-group;

Clearing of the next signal group.

In one embodiment, three signal groups are employed for list-formcommands. In the case of list-form commands, the three signal groups areemployed for the following reasons:

-   -   The current entry for the entry being operated on;    -   The RTC for the current entry is received and followed by the        RTC for the next entry before the current signal group index is        incremented;    -   The previous entry, in the event that the MRB for the received        RTC is lost and a subsequent RTC is returned for the last entry,        as a result of the timeout at the other coupling facility.

Execution of a Duplex-Signal List-Notification Command

The duplex-signal list-notification command provides the informationused by the designated system to update one signaling-vector entry bit.

Execution of a list-notification command involves first selecting amessage path from the set of message paths that form an activeconnection with the remote coupling facility and making the listnotification command pending on this path.

When a pending list-notification command is executed, the command isissued to the duplex coupling facility.

Execution of a duplex-signal list-notification command is completed,when a message-response block is received at the coupling facility inresponse to the command. Duplexing is broken when, for instance: (1) allmessage paths in the path group are inactive at the time of pathselection or (2) the state of all of the active message paths to theassociated system is made error state pending by the system and inactiveby the coupling facility.

Duplex-signal list-notification commands are ensured to complete, inthis embodiment.

Notes on the Execution of a Duplexing Signal

-   -   1. When a duplex-signal list-notification command cannot be        successfully delivered to a system (e.g., the message times        out), the coupling facility attempts to deliver the command over        each of the remaining active message paths to the system until a        message-response block is received or all paths in the path        group have been made error state pending by the system and        inactive by the coupling facility.    -   2. Broadcasting the same list-notification command on multiple        paths in the path group allows for faster completion of the        list-notification process when link errors are present. However,        the broadcast protocol should be limited to situations where the        error-state-pending condition has been set after an unsuccessful        attempt to send a single list-notification command for the        transition. Broadcasting in non-error situations could add        significant load to the link and should be avoided. Broadcasting        the same list-notification command on multiple paths is allowed,        but the signal-group index is not incremented by one until such        time as all list-notifications commands in the broadcast have        completed or been suppressed.    -   3. When the path group is empty at the time of path selection,        or when all paths in the path group are inactive, the generated        list-notification command completes without initiating a link        operation. If the duplexing signal is RTE, the command is        completed and a response code (e.g., 253) is returned.        Otherwise, duplexing is broken.    -   4. The signal-vector token is assigned by the IMP command and        returned in the MRB for the IMP command. The coupling facility        provides the signal-vector token in the list-notification        command. The system uses the signal-vector token to determine        the location of the signal vector in processor storage.

Duplex Command Timeouts

When the duplex command times out and an invalidate request is receivedat the coupling facility, if (1) the ready-to-execute signal has notbeen sent, then the response code is set to, for instance, 253, and theinvalidate response is sent and the command is suppressed; if (2) theready-to-execute signal has been sent, and the ready-to-execute orready-to-complete has not been recognized, then a halt-execution signalis sent, the response code is set to, for instance, 253, and theinvalidate response is sent and the command is suppressed; if (3) theready-to-execute signal has been sent and the ready-to-execute orready-to-complete has been recognized, the invalidate response is sent,and the command is allowed, for example, 300 milliseconds after theready-to-execute or the first ready-to-complete signal was recognized tocomplete. Otherwise, the command is suppressed, when the 300milliseconds expires setting the duplexing inactive bit, and a responsecode of, for instance, 20 (duplexing inactive). During the 300millisecond window, no new commands are executed at the couplingfacility, and the duplex command continues to execute. After theadditional 300 milliseconds has transpired, the command is commandsuppressed and new commands can then be executed in the couplingfacility.

Notes on Duplex Command Timeouts:

-   -   1. It is possible for a duplex command to be delayed in the        decode of the command, resulting in a duplex command timeout,        while the command is in the process of executing. In order to        avoid the chance that duplexing is broken, an additional 300        milliseconds is added to allow the command time to execute and        complete. The suspension of new commands decoding at the        coupling facility allows the normal coupling command recovery        actions to take place without the chance that a subsequent        command has compromised the integrity of the objects in the        coupling facility.    -   2. These actions reduce the likelihood that duplexing will be        broken. Breaking of duplexing is an action that is to be        avoided, if possible.

Breaking of Duplexing

Duplexing is broken, when the objects in the two coupling facilities arenot synchronized.

When a duplexing command is executing and the command ends in a couplingfacility, a ready-to-complete signal has been sent, and the entry is notcommitted or backed out, then the duplex active bit for the SID in theduplexing vector is set to zero. If the command is a single-list entrycommand, the command is completed and the MRB is returned. If thecommand is a list-form command, processing of the current list item iscompleted, processing of the command is completed at the current listitem, and the MRB is returned. In either case, the duplexing-deactivatedindicator is set to B′1′ in the response descriptor.

If a duplex command attempts to execute and there are no paths availablebetween the two coupling facilities, then the duplex active bit for theSID in the duplexing vector is set to zero, the command is suppressedand a response code (e.g., 20) is returned.

When a duplex command is executed and the duplex active bit for the SIDin the duplexing vector is zero, the command is suppressed and aresponse code (e.g., 20) is returned.

Notes on Breaking Duplexing:

-   -   1. The breaking of duplexing design allows the coupling channels        that connect duplexed coupling facilities to lose connectivity        and recover connectivity without breaking duplexing, so long as        a duplex command is not being executed during the time that        connectivity is lost.        Implementation of Coupling Facility Duplexing Signaling Protocol

Further details regarding one implementation of a duplexing signalingprotocol is described below.

Commands that are duplexed between coupling facility structures use peersignaling, in accordance with an aspect of the present invention, tokeep the commands in synchronization. There are three steps that thesignals keep in synch:

-   -   The Exchange of the Ready-to-Execute (RTE) signal;    -   The Latching of Resources required to complete the command;    -   The Exchange of the Ready-to-Complete (RTC) signal.

The latching of resources step presents a problem since many commands,both simplex and duplex, may be competing for the same resources. Forone coupling facility structure, Command A may win the race to latch aresource, but for the duplexed structure, Command B may win the race forthe same resource. This implementation resolves conflicts in latchingresources by using a combination of techniques, all of which includecommunications between tasks within a structure. A Task Control Block(TCB) is used as a communications medium. Each task has its own TaskControl Block and is able to access and update the Task Control Block ofthe other tasks. Each update of an item in the TCB is done atomically,so that no partial updates occur.

In one example, the Task Control Block (TCB) includes, for instance:

-   -   A command sequence number (CSN) used to assign a priority to        each duplexed command. (The smaller the CSN, the higher the        priority. The CSN for a simplex command is 0, the highest        priority).    -   A Suppress Command flag is used as means of resolving conflicts.        A command which has received an RTC signal, but is waiting for a        resource, and determines its CSN has higher priority than the        resource holder's CSN, sets this flag in the TCB of the holder        of the desired resource. When a task sees this flag set in its        TCB, while it is waiting for an RTC signal, it requests        suppression for the command in the duplexed structure by sending        a Request for Suppression (RFS) signal to the duplexed        structure. Additionally, it suppresses its command, when the        request for suppression is acknowledged via a Request for        Suppression Acknowledgment (RFSA) signal from the duplexed        structure.    -   A Proxy RTC flag is used to indicate a duplexed command has        received a Ready-to-Complete signal and is waiting for a        resource. The command which has received the RTC signal does not        have priority to the resource, but the holder of the resource        may be a simplex command which does not participate in the        signaling, and it may be waiting for a resource held by a lower        priority duplexed command. The proxy RTC flag also indicates        that there is a proxy CSN that should be compared with the CSN        in the TCB of any latch holders of a resource in a chain of        commands in which each is waiting for some resource. The task        which has its Proxy RTC flag set compares its proxy CSN with the        CSN of the task holding a resource for which it is waiting.    -   A Proxy CSN is stored in the TCB of a holder of a resource        whenever the Proxy RTC flag is set.

Further details regarding the signaling protocols employed in one ormore aspects of the present invention are described with reference toFIGS. 16 a-21. In particular, FIGS. 16 a-16 b provide an overviewcontrol flow of duplexed commands, and FIGS. 17, 18, 20-21 provideembodiments of control flows for various functions invoked in theoverview and/or used by the protocol. FIG. 19 provides an extension oflatch management for simplex commands, which intersects with thiscontrol flow. In one example, the logic of FIGS. 16 a-21 are executed bythe coupling facility.

Referring to FIG. 16 a, initially, a command is received by the CFCC anda task control block (TCB) is assigned to the command, STEP 1600. Also,the CSN for the task is zeroed. Next, a duplex command decode checkingfunction is invoked, STEP 1602. (This is described further below withreference to FIG. 17.) Should one of the checks of the decode checkingfunction fail, INQUIRY 1604, the command is suppressed and theappropriate response code is returned, STEP 1606. If, however, all ofthe checks are successful, then an RTE exchange function is invoked,STEP 1608. (This is described further below with reference to FIG. 18.)Should the RTE exchange fail, INQUIRY 1610, the command is suppressedand a response code (e.g., 253) is returned, STEP 1606. If, however, theRTE exchange is successful, then the command is executed, STEP 1612.During execution, a check is made as to whether any halting conditionshave been recognized, STEP 1614.

If any halting conditions are encountered during command execution,INQUIRY 1616, then a halt signal is sent, STEP 1618, and the commandcompletes with the response code appropriate for the halting conditionthat was encountered, STEP 1606. Otherwise, a duplex latch resourcefunction is invoked for each resource that needs to be latched by thecommand for processing of the current list item, STEP 1620 (FIG. 16 b).(This is described in further detail with reference to FIG. 20.) If alllatch obtains are successful, INQUIRIES 1622, 1624, then the commandcompletes processing of the current list item and the RTC exchangefunction is invoked, STEP 1626. (This is described in further detailwith reference to FIG. 21.) However, if a latch is not successfullyobtained, then the command ends and a response code is returned, STEP1606 (FIG. 16 a).

Returning to FIG. 16 b, should the RTC exchange function end withoutreceiving an RTC signal, INQUIRY 1628, the command is completed and aresponse code is returned step, 1606 (FIG. 16 a). If, however, an RTCsignal is received, the processing of the current list item iscompleted, STEP 1630 (FIG. 16 b). If the command is a single-list entrycommand, INQUIRY 1632, the MRB is returned and the latches are released.In this case, processing is complete, STEP 1606 (FIG. 16 a). If thecommand is a list-form command and either a model-dependent timeout isexceeded or the last list item is processed, INQUIRY 1634 (FIG. 16 b),then, the command is completed with either an RC=1 or RC=0, asappropriate; the MRB is returned and the latches are released. Again,processing for the command is completed, STEP 1606 (FIG. 16 a). If thecommand is a list-form command, the current list item is not the lastlist-item, and a model-dependent timeout has not been reached, thelatches are released and the next list item is processed, STEP 1620(FIG. 16 b). This completes a description of one embodiment of anoverview of duplex signal processing.

FIG. 17 describes one embodiment of the set of checks performed duringthe command decode phase of a duplexed command. Initially, a duplexcommand is received, STEP 1700. Then, the first check is made to ensurethat a valid SID is specified, INQUIRY 1702. If not, the command hasspecified an invalid structure. Thus, the command is suppressed and an‘invalid SID’ response is returned, STEP 1704. If, however, the SIDspecifies a valid structure, then the command sequence number (CSN) istested, INQUIRY 1706.

If the CSN is zero, the command is not executed as a duplexed command.It may be executed as a simplex command or not executed at all based onthe setting of the DUPAI indicator and the current state of thestructure. For instance, if the DUPAI indicator is B′0′, INQUIRY 1708,the command is executed as a simplex command, no matter the structurestate, STEP 1710. If the DUPAI indicator is B′1′ and the structure is inthe duplexing active state, the command is also executed as a simplexcommand, STEP 1710. However, if the DUPAI indicator is B′1′ and thestructure state is duplexing inactive, the command is suppressed and aresponse code (e.g., 20) is returned, STEP 1712.

(Notes on CSN and DUPAI) Setting the CSN indicator to zero is a meansfor the operating system to execute a command in simplex mode, when thestructure is duplexing active. This is done, for instance, for aRead-and-Register command where the storage class is not being changedby the command. Since the data is only returned from the primary cachestructure and since the registrations are only made in the primarystructure, there is no reason to send the command to the secondarystructure. So, the command is issued with the CSN indicator set to B′0′.This is done for various commands that are reading data or controls inthe primary structure, but do not update any control objects that areduplexed.

However, a window exists for read commands that set the CSN to zero andare issued after duplexing has failed, but before the structure hasentered simplex mode. During the recovery window, the structure objectsmay be out of synchronization. So, a read command sent to one structurewith CSN=0 may execute successfully and observe object states that maynot exist after the simplex mode is resolved. For instance, a Move ListEntry command may move a list entry in the secondary structure, but notthe primary structure. If a Read List Entry command is issued to thesecondary structure during structure failover with CSN=0, the moving ofthe list entry may be observed. However, if the primary structure isselected, the moving may be suppressed by the failover. The observing ofa moved list entry that is not in fact moved is a violation of commandconcurrency. This problem is avoided by setting the DUPAI indicator toB′1′ along with setting the CSN to zero. Then, if the structure hasbecome duplexing not active, the read command is suppressed.

Continuing with FIG. 17, if the command sequence number is non-zero, arequest is made to execute the command using the duplexed commandprocesses. So, the structure state is checked to see if duplexing isactive for the structure, INQUIRY 1714. If duplexing is not active, thecommand is suppressed and a response code (e.g., 20) is returned, STEP1712. If duplexing is active, a test is made to see if the remotefacility is connected, STEP 1716. If the remote facility is notconnected, the command is suppressed and a response code (e.g., 253) isreturned, STEP 1718.

If the remote facility is connected, the state of the structure is againtested to see if dumping serialization is held, STEP 1720. If so, thecommand is suppressed and the ‘dumping serialization held’ response isreturned, STEP 1722. If dumping serialization is not held, all the testsare successfully completed, and the processing of the duplexed commandcontinues, STEP 1724.

FIG. 18 describes one embodiment of the logic associated with theready-to-execute (RTE) exchange function. The function is invoked toboth send an RTE signal to the remote coupling facility and to recognizethe reception of an RTE signal. The individual steps are as follows.First, an RTE signal is sent to the remote coupling facility on one ofthe receiver ISCs identified in the path group for the remote couplingfacility, STEP 1800. The signaling command designates the signalingvector entry in the remote coupling facility associated with theduplexed command. Next, the task control block (TCB) assigned to thecommand is initialized by saving the command sequence number (CSN)specified in the MCB into the TCB, STEP 1802, and by resetting the‘suppress command’ flag and the ‘Proxy RTC’ flag, which are in the TCB,STEP 1804.

Next, there is a check to see if an RTE signal has been received fromthe remote coupling facility in the signaling vector entry associatedwith the command, INQUIRY 1806. If an RTE signal has been received, thecommand timer is started and the return code for the function is set to‘success’, STEP 1808. On the other hand, if an RTE signal has not beenreceived, the primary and secondary links are tested to see if anyerrors have occurred that would have deactivated the links, INQUIRY1810. The primary link is the link where the RTE signal from the remotecoupling facility is received and the secondary link is the link wherethe RTE signal is sent. If there are no active primary or secondarylinks, the signal exchange is aborted and the function ends with returncode of ‘failure’, STEP 1812. If there are still active links, thefunction loops back to the check of reception of the RTE signal, INQUIRY1806.

(Note on RTE exchange.) The RTE signal is employed to improve bothperformance and reliability of the protocol. By delaying the start ofcommand execution until both commands have been received and recognizedby their respective coupling facilities, the latch hold times, and anyresulting contention, are minimized. This is especially desirable if thedistances between the coupling facilities are large and the commands areskewed in time by the propagation delays involved. Therefore, includingthe RTE exchange in the protocol improves the performance of theprotocol by minimizing the latch hold times.

A second reason for including the RTE exchange is for improvedavailability. The protocol indicates that duplexing is to be broken bythe coupling facility if a completion signal (RTC) has been sent to theremote coupling facility, but no completion signal has been receivedbefore the command times out. But, this rule does not apply to the RTEexchange, since no object updates have yet occurred. A failure conditionin the RTE exchange results in the command being suppressed and aresponse code (e.g., 253) being returned. In this case, the duplexingstate does not change. If the failure is temporary in nature, then thecommand can be redriven by the operating system for a relatively shortduration of say a few seconds, and the temporary condition may berectified in that period of time. If the RTE exchange is not made, thenany temporary error detected by the RTC exchange causes duplexing to bebroken. By requiring a successful exchange of signals prior to sending acompletion signal, the chances of duplexing being broken are greatlyminimized, and the resulting protocol is more robust.

(Notes on duplexing command deadlocks.) A basic deadlock scenario withduplexed commands can occur as follows. Suppose Command A is issued byOS 1 and Command B is issued by OS 2 and the two commands attempt tolatch the same resources. A deadlock occurs if Command A is executed onCF 1, obtains the latch on the resource and issues an RTC signal.Command B on CF 1 now waits for Command A to complete before getting thelatch. Command A is waiting on an RTC from its duplexed instance runningon CF 2. However, the situation on CF 2 is reversed, and Command B hasobtained the latch and sent the RTC signal and Command A is delayed.This is the deadlock. The deadlock is resolved by comparing the commandsequence numbers for the two commands. If Command A has a lower (i.e.,higher priority) CSN than Command B, then the latch manager (which is acomponent of CFCC) on CF 2 forces Command B to be suppressed. Thiscauses Command B to issue a request for suppression to CF 1. This willbe accepted in this circumstance and a request for suppression accepted(RFSA) will be sent by CF 1. Command B then backs out on CF 1, andCommand A can then obtain the resource and the duplexed pair for CommandA completes. Meanwhile, OS 2 sees the suppression of Command B andreissues the command with the same CSN. Since the CSN is an increasingvalue across all the commands, the priority of Command B will eventuallyexceed the priority of all other commands and be assured to complete.

In a further aspect, the latch manager also detects deadlocks that canoccur when intervening commands, including simplex commands, holdlatches. In this case, the proxy controls in the TCB are used torecognize chain conditions and impose the above protocol even whenintervening commands are present. This is described in FIGS. 19 and 20.Two more complex scenarios are described below after FIG. 21 isdescribed.

FIG. 19 describes one embodiment of the actions taken to latch aresource by a simplex command for structures where duplexing is active.The function begins by attempting to latch the resource, STEP 1900. Ifthe resource is successfully latched, INQUIRY 1902, then the proxy RTCflag is reset in the TCB and the function ends with a return code of‘success’, STEP 1904. Otherwise, the proxy RTC flag is tested, STEP1906.

If a proxy RTC signal was not received, INQUIRY 1908, processing loopsback to reattempt to latch the resource, STEP 1900. If, on the otherhand, a proxy RTC signal was received, then a comparison is made betweenthe task proxy CSN and the CSN of the latch holder, STEP 1910. If thetask proxy CSN is a higher priority (lower value) than the latch holderCSN, INQUIRY 1912, then the suppress command flag is set in the latchholder's TCB, STEP 1914, and processing loops back to reattempt to latchthe resource, STEP 1900. Setting the suppress command flag causes thelatch holder to release the latches, when it detects the flag has beenset.

If, on the other hand, the task proxy CSN is not of a higher priority,then a test is made to see if the latch holder received a proxy RTCsignal, INQUIRY 1916. If not, then the latch holder task's TCB isupdated by storing this task's CSN as the proxy CSN and by setting theproxy RTC flag, STEPS 1918, 1920. Then, processing loops back toreattempt to latch the resource, STEP 1900.

However, if the latch holder did receive a proxy RTC signal, then acomparison is made between this task's proxy CSN and the latch holder'sproxy CSN, INQUIRY 1922. If this task's proxy CSN has priority, INQUIRY1924, then the latch holder's proxy CSN is replaced with this task'sproxy CSN, STEP 1926, and processing loops back to reattempt to latchthe resource, STEP 1900. If this task's proxy CSN does not havepriority, no updates are made and processing loops back to reattempt tolatch the resource, STEP 1900.

FIGS. 20 a-20 b describe one embodiment of the duplex latch resourcefunction. Processing begins in FIG. 20A with an attempt to latch theresource, STEP 2000. If latching is successful, INQUIRY 2002, then theproxy RTC flag in the TCB is reset and the function ends with a returncode of ‘success’, STEP 2004. However, if latching was not successfulthen the signaling vector is tested to see if one of three possiblesignals was received from the remote coupling facility: request forsuppression, halt or an RTC. Testing is performed in that order, as anexample.

If a request for suppression signal was received, INQUIRY 2006, then arequest for suppression accepted signal is sent and the function endswith ‘request for suppression’, STEP 2008. The command will besuppressed in this case.

If a halt signal was received, INQUIRY 2010, then the function ends withan ‘execution halted’ condition, STEP 2012. Again, the command will besuppressed, but no signals need to be sent in this case.

If, on the other hand, an RTC signal was received, INQUIRY 2014, acomparison is made of this task's CSN with the latch holder's CSN, STEP2016. If this task has priority, INQUIRY 2018, then the suppress commandflag is set in the latch holder's TCB, STEP 2020, and processingcontinues in FIG. 20 b at label (A). If this task does not havepriority, but the latch holder has received a proxy RTC signal, INQUIRY2022, then this task's CSN is compared to the latch holder's proxy CSN,STEP 2024, and processing continues in FIG. 20 b at label (B).

If the latch holder did not receive a proxy RTC signal, then the latchholder's TCB is updated by storing this task's CSN in the proxy CSNfield and setting the proxy flag, STEP 2026. Processing then continuesin FIG. 20 b at label (A). If no RTC signal has been received, INQUIRY2014, then processing continues in FIG. 20 b at label (A).

Label (B) in FIG. 20 b completes the tests for when an RTC signal hasbeen received. At label (B), a test is performed to see if this task'sCSN has priority over the proxy CSN of the latch holder's task, INQUIRY2030. If not, then processing continues at label (A). If so, then thelatch holder's proxy CSN is replaced with this task's CSN, STEP 2032,and processing continues at label (A).

Label (A) in FIG. 20 b resumes the duplex command function. First, atest is made to see if the suppress command flag has been set in thistask's TCB, INQUIRY 2034. If so, then a ‘Halt’ signal is sent to theremote coupling facility, and the function ends with a response that ahalt signal was sent, STEP 2036. However, if the suppress command flagis not set, then the proxy flag is tested, INQUIRY 2038. If no proxysignal was received then processing loops back to the top of FIG. 20 aat label (C) where a reattempt is made to latch the resource. If, on theother hand, a proxy RTC signal was received, then a comparison is madeof this task's proxy CSN with the CSN of the latch holder, STEP 2040. Ifthis task's proxy CSN has priority, INQUIRY 2042, then the suppresscommand flag is set in the latch holder's TCB, STEP 2044, and processingresumes at label (C). If this task's proxy CSN does not have priorityover the latch holder's CSN, a test is made to see if the latch holderreceived a proxy RTC signal, INQUIRY 2046. If not, then the latchholder's TCB is updated by storing this task's CSN as the proxy CSN andsetting the proxy flag, STEP 2048. Processing then resumes at label (C).However, if the latch holder did receive a proxy RTC signal, then acomparison is made to see if this task's proxy CSN has priority of thelatch holder's proxy CSN, INQUIRY 2050. If so, then the proxy CSN in thelatch holder is replaced with this task's proxy CSN, STEP 2052, andprocessing continues at label (C). If not, no updates are made andprocessing resumes at label (C).

FIG. 21 describes one embodiment the RTC exchange function. Once thelatches have been obtained for the objects associated with the command,the updates to the objects have been made, and the MRB is ready to besent, the RTC exchange function is invoked. The exchange sequence beginswith the sending of an RTC signal, STEP 2100. Next, a test is performedto see if an RTC signal has arrived, INQUIRY 2102. If so, then thefunction ends with a success indication, STEP 2104. Otherwise, a test isperformed to see if a halt signal has arrived, INQUIRY 2106. If so, thenthe function ends with an indication that execution was halted, STEP2108. If not, a test is performed to see if the suppress command flaghas been set in the task control block, INQUIRY 2110.

If the flag is not set, the checks for the signal reception is continueduntil either a signal is received or the command timer expires, INQUIRY2111. If the suppress command flag is set, then the latch manager hasdetermined that a latch held by this command may be creating a deadlocksituation and the priority decision is for this command to back out.This is done as follows. First, a request for suppression signal is sentto the remote coupling facility to inform that coupling facility of thepotential deadlock and the need to suppress this command, STEP 2112. Thefunction then waits on reception of a signal up to the point when thetimer expires. If an RTC signal is received, INQUIRY 2114, then thecommand can complete normally. No deadlock exists and normal completionwill free the necessary latches. So, the function ends with a successindication, STEP 2116. If an RTC signal is not received, but a haltsignal is received, INQUIRY 2118, then the function ends with anindication that execution was halted, STEP 2120.

If a request for suppression accepted signal is received, INQUIRY 2122,then the other coupling facility has acknowledged the request forsuppression signal and the function ends with this indication, STEP2124. If no signal of any kind is received and the command timerexpires, INQUIRY 2126, duplexing is broken, STEP 2128. The duplexingactive indicator for the structure in the duplexing vector is set toB′0′ and the function ends with a duplexing inactive indication, STEP2130.

(Note on breaking duplexing.) Once an RTC signal has been set, thecoupling facility has committed to completing the command. If itreceives an RTC signal or an RFS signal, then it can safely complete thecommand. It can also back out the command, if it receives a halt signalor if it sends an RFS signal and receives an RFSA signal. In eithercase, the protocol rules ensure that both sides suppress the command.However, if no signal of any kind is received and the command timerexpires, duplexing is to be broken, since there is no positiveindication from the other coupling facility that it has either completedthe command or suspended the command. Ending the command without thispositive acknowledgment is not allowed without also breaking duplexing.Otherwise, the structure states may be different once the latches aredropped. This violates the desire to keep the two structures in completesynchronization.

Deadlock Avoidance Scenarios:

In the follow scenarios, Command n has a command sequence number withvalue n. So, Command 1 is a higher priority than Command 2, and Command2 has a higher priority than Command 3. The latch manager detects thepotential deadlock situations and uses a combination of signals (RFA,RFSA, and Halt) and TCB flags (suppress command and proxy RTC) toresolve these deadlocks. The logic is described in the FIGS. 18, 20, and21 and the following examples serve only to illuminate this logic withexplicit examples. Each case can be reduced to either the basic deadlockscenario described above or one of these three cases.

Deadlock with Intervening Lower Priority Command:

In this scenario, three commands, Commands 1, 2 and 3, are issued bythree separate systems to a pair of duplexed structures: one residing onCF 1 and the second residing on CF 2. Commands 1 and 3 need to latchResources x and y and Command 2 only needs to latch Resource x. Thehierarchy of latching rules in the coupling facility requires if both xand y need to be obtained, y is obtained first.

The order of arrival and subsequent execution is as follows: On CF 1,Command 1 obtains latches for both x and y and sends an RTC signal to CF2. Commands 2 and 3 both wait in the latch manager on CF 1. Command 2waits on latch x and Command 3 waits on latch y. On CF 2, Command 2executes first and obtains the latch on Resource x and sends an RTCsignal to CF 1. Command 3 obtains the latch on Resource y and waits onthe latch on Resource x. Command 1 waits on latch y in the latch manageron CF 2.

A deadlock exists between Commands 1 and 2. Command 1 has obtained allits latches on CF 1, including the latch on x that Command 2 needs.Meanwhile, Command 2 has obtained the latch on x, which Command 1 needson CF 2. Also, Command 1 on CF 2 has received an RTC signal from CF 1.However, this differs from the basic deadlock case, since a thirdcommand, Command 3 owns the latch that Command 1 is requesting first.Command 3 is an intervening command of lower priority than Command 1.

This more complex scenario is resolved as follows: The latch managerservicing the request for the latch on y by Command 1 determines thatthe latch holder has lower priority and sets the suppress command flagin the TCB for Command 3. Next, when the latch manager services therequest by Command 3 for latch x, the latch manager detects that thesuppress command flag is set for Command 3. But, no RTC signal has yetbeen sent for Command 3, since not all latches have been obtained. So,the latch manager issues a ‘Halt’ signal to CF 1 and Command 3 completeswith a ‘command halted’ response. Likewise, when the latch manager on CF1 services Command 3's request for the latch on y, it detects thereception of a halt signal and ends the command with a ‘command halted’response.

Once Command 3 has been halted, Command 1 can obtain the latch on y onCF 2 and then detect contention with Command 2 on latch x. The situationhas reduced to the basic deadlock, and is resolved as described above.

Deadlock with Intervening Higher Priority Command:

In this scenario, three commands, Commands 1, 2 and 3, are issued bythree separate systems to a pair of duplexed structures; one residing onCF 1 and the second residing on CF 2. Commands 1 and 2 need to latchresources x and y and Command 3 only needs to latch resource x. Thehierarchy of latching rules in the coupling facility requires if both xand y need to be obtained, y is obtained first.

The order of arrival, and subsequent execution is as follows: On CF 1,Command 2 obtains latches for both x and y and sends an RTC signal to CF2. Commands 1 and 3 both wait in the latch manager on CF 1. Command 1waits on latch y and Command 3 waits on latch x. On CF 2, Command 3executes first and obtains the latch on Resource x and sends an RTCsignal to CF 1. Command 1 obtains the latch on Resource y and waits onthe latch on Resource x. Command 3 waits on latch y in the latch manageron CF 2.

A deadlock exists between Commands 2 and 3. Command 2 has obtained allits latches on CF 1, including the latch on x that Command 3 needs.Meanwhile, Command 3 has obtained the latch on x, which Command 2 needson CF 2. Also, Command 2 on CF 2 has received an RTC signal from CF 1.However, this differs from the basic deadlock case, since a thirdcommand, Command 1 owns the latch that Command 2 is requesting first.Command 1 is an intervening command of higher priority than Command 2.Also, while Command 1 has higher priority than Command 3, it does notsuppress Command 3 to obtain latch x, because no RTC signal has beenreceived by Command 1.

This more complex scenario is resolved as follows: The latch managerservicing the request for the latch on y by Command 2 determines thatthe latch holder has higher priority and sets the proxy RTC flag and theproxy CSN in the TCB for Command 1. Next, when the latch managerservices the request by Command 1 for latch x, the latch manager detectsthat the proxy RTC flag is set for Command 1. It then compares the proxyCSN (=2) in the TCB for Command 1 against the CSN for Command 3 (=3). Itdetermines that the proxy task has priority and sets the suppresscommand flag in the TCB for Command 3. Command 3 is waiting on the RTCsignal from CF 1 and in this function detects that the suppress commandflag is set. It then sends a request for suppression signal to CF 1 andwaits for the RFSA or RTC or Halt to be received. CF 1 sees the requestfor suppression of Command 2 and, since Command 2 is waiting on aresource, a request-for-suppression accepted signal is sent and Command2 is suppressed on CF 1 with a ‘command suppressed’ response. When theRFSA signal is received on CF 2 for Command 3, Command 3 releases thelatches on x and y and completes the command with a ‘command suppressed’response. Command 1 then obtains the latches on y and x and sends an RTCsignal. There is now a deadlock between Commands 1 and 2, which is thebasic deadlock scenario which is resolved, as described above. It isinteresting to note that the basic deadlock is resolved by the latchmanager on CF 1. That is, the first deadlock is resolved on CF 2 and thesecond deadlock is resolved on CF 1.

Deadlock with Chain of Intervening Higher Priority Commands:

This third example is a more complicated version of the previous exampleand shows how the proxy RTC is passed across a chain of commands. Inthis scenario, n+1 commands, Commands 1, 2 . . . n+1, are issued by n+1separate systems to a pair of duplexed structures; one residing on CF 1and the second residing on CF 2. All the commands need a command latchon Resource x. Moreover, there is a sequence of resources labeled y1, .. . , yn. Command 1 needs just y1. Command 2 needs both y1 and y2. Thispattern continues with Command i needing y(i−1) and y(i). The hierarchyrules are that for each pair of integers i<j, yy is obtained before yiand all the y resources are obtained before Resource x.

The execution sequence is as follows: On CF 1, Command n has obtainedlatches y(n−1), yn and x, and has sent an RTC signal. All the othercommands are waiting on latches. On CF 2, Command n+1 has obtained latchx and has sent an RTC signal. So, the deadlock is between Commands n andn+1. But, the latching in CF 2 is as follows: Command 1 Command n isreached, which has obtained yn and is waiting on y(n−1). Command n isthe only command that has received an RTC signal, so it starts the proxysequence by setting the proxy flag and proxy CSN in the TCB for Commandn−1, which in turn propagates these values to Command n−2, etc. In eachcase, command suppression is not set because the current command is of ahigher priority than Command n. That is, until Command 1 tests Commandn+1. In this case, the proxy CSN (=n) is higher priority than the CSNfor Command n+1 (=n+1), and so, processing for Command 1 sets thesuppress command flag in the TCB for Command n+1 and Command n+1 sendsan RFS signal. Once Command n+1 has been suppressed and releases thelatch on x, Command 1 now obtains Resource x and sends an RTC signal.This now creates a deadlock between Command 1 and Command n, which isdetected and resolved by the latch manager on CF 1. Once that deadlockis resolved, the commands can complete in priority order.

Duplexing of Cache and List Commands

Response-Descriptor Fields

The duplexing signaling protocol described above is invoked whencommands, such as cache and list commands, are executed as duplexedcommands. Examples of commands that can be invoked as duplexed commandsare described below.

For each of the commands, an MRB is returned, which has aresponse-descriptor. One embodiment of the response descriptor fieldsreturned for a duplexed command is described below.

Two fields in the response descriptor, the response count and the datacount, define the number of meaningful bytes of information that arereturned in the MRB and the data block, respectively. Two additionalfields are associated with structure duplexing: the current signal-groupindex is used to coordinate the signaling protocol between couplingfacilities that contain duplexed structures; and theduplexing-deactivated indicator is used to alert the program that theduplexing state was changed from active to inactive, during commandexecution. Each of these fields is further described below.

Response Count: The value of the response count is the number ofmeaningful bytes returned in the MRB as described in the MRB format foreach command. Reserved or unused bytes at the end of the MRB may beexcluded.

Data Count: When a data block is returned to the program, the data countis set to the number of, for instance, 256-byte increments returned bythe coupling facility. Reserved or unused bytes at the end of the datablock may be excluded. The value of the data count times 256 is to besmaller than or equal to the message-buffer size, in bytes.

Current Signal-Group Index (CSGX): When duplexing signals are issued fora command, the value of the current-signal-group-index object in thesignaling-vector entry associated with the retry index is stored in, forinstance, bits 30-31 of word 2. If duplexing signals are not generatedfor the command, zeros are stored in bits 30-31 of word 2.

Duplexing-Deactivated Indicator (DDI): The value of theduplexing-deactivated indicator describes the result of the single-entryor list-form duplexing process for duplexed commands. When theduplexing-deactivated indicator is, e.g., B′1′, duplexing was brokenduring command execution and the duplexing-active bit was reset in theduplexing vector. When the duplexing-deactivated indicator is B′0′,either the duplexing process completed successfully, or no duplexingprocess was executed for the command. The duplexing process completessuccessfully, when the duplexing signaling protocol is completed and therequired signals are received. The value of the duplexing-deactivatedindicator is set to B′0′, when the duplexing command completes with aresponse code 20, as one example.

Notes on the DDI and CGSX Operands:

-   -   1. Duplexing signals are generated for any command that        satisfies the following criteria: (a) The command is issued for        a structure for which the duplexing controls are active. (b) The        MCB includes a non-zero command sequence number. (c) The command        completes with a response code less than 254 (as an example) or        completes with the following status conditions, as examples:        4—Request exception, except invalid SID, or 7—Dumping        serialization held.    -   2. If an invalid SID is specified, no signals are issued, zeroes        are stored in bits 30-31 of word 2 of the response descriptor,        and a value of 3 is stored in the exception code.    -   3. When a non-zero current-signal-group index is stored in the        response descriptor, the associated signal group is reset.    -   4. When the value of the current signal-group index is zero, no        signals were sent to the remote coupling facility and no signal        groups in the signaling-vector were reset. However, signals may        have been received from the remote facility. The program        inspects the CSGX values returned in both MRBs. If both are        zero, then no signals were issued by either coupling facility        and the old CSGX values can be reused. If one of the returned        values is zero and the other non-zero, the signaling-vector        entry in the coupling facility that returned a zero-valued CSGX        is reset. Resetting the signaling-vector entries in both        coupling facilities is also acceptable.    -   5. The duplexing-deactivated indicator may be set to zero during        the command decode phase before sending the RTE signal. If the        command completes without breaking duplexing, no additional        update needs to be made. However, if duplexing is broken during        command execution, then the duplexing-deactivated indicator is        set to, for instance, B′1′, while the objects in the structure        that are referenced by the command are serialized.        Cache-Structure Operands

In addition to the operands described above. Each cache structure hasits own operands that are associated with the cache, when it is created.Examples of these operands are described below.

Comparative Structure Authority (CSAU): A value used as a comparisonvalue to the structure authority, when the structure is allocated anddeallocated, or when castout locks are reset, the detachment-emulationcontrol (described below) is B′1′, and duplexing is active.

This operand is ignored on an Unlock-Castout-Locks command, whenduplexing is inactive, or when the detachment-emulation control is B′0′.

Comparative Remote-Facility Structure Authority (CRFSAU): A value usedas a comparison value to the remote-facility structure authority, whencastout locks are reset, the detachment-emulation control is B′1′, andduplexing is active.

This operand is ignored, when duplexing is not active, or when thedetachment-emulation control is B′0′.

Command Sequence Number (CSN): A value associated with a command that isduplexed. Cache-structure commands that specify a non-zero value in theCSN request operand cause the invocation of the duplexing-commandprocess, when duplexing is active for the structure. Commands that donot have the CSN request operand defined, or which specify a zero valuein the CSN request operand, do not invoke the duplexing-command process.

Detachment-Emulation Control (DTEMC): A value that controls theprocessing of the Unlock-Castout-Locks command. The two possible valuesare:

0 Normal command execution; 1 Detach-processing rules are used.

When the detachment-emulation control is B′1′, the change bitoverindication (CBO), castout parity bits indicator (CP) and user datafield (UDF) operands in the unlock-castout-locks (UCL) items and thecastout-process identifier request operand are not used and are ignored.

Directory Position (DIRP): A value that denotes a position of adirectory entry in the directory.

Duplexing-Active Indicator (DUPAI): A value that controls execution ofthe command based on the duplexing state of the structure. It has thefollowing encoding:

0 Do not test the duplexing state; 1 Test the duplexing state of thestructure.

Duplexing Signal-Group Index (DSGX): A value that identifies the targetsignal group in the signaling-vector entry identified by the duplexingretry index and the remote-facility controls. If duplexing is active andthe command sequence number is non-zero, the duplexing signal-groupindex is non-zero. If duplexing is not active for the structure or ifthe command sequence number is zero, the operand is ignored.

Duplexing Retry Index (DRX): A value that designates a signaling-vectorentry for the signaling vector identified by the remote-facilitycontrols. If duplexing is active and the command sequence number isnon-zero, the duplexing retry index is non-zero. If duplexing is notactive for the structure or if the command sequence number is zero, theoperand is ignored.

Failed-Structure Indicator (FSI): A value that controls the state changefor the structure that occurs when a Deallocate-Cache-Structure commandis executed and the structure is in the allocated state. It has thefollowing encoding:

0 Initiate deallocation of the structure; 1 Place the structure in thestructure-damage state.

Immediate Reclaim Control (IMMRC): A value that determines what reclaimaction should occur when the castout lock is reset. It has the followingencoding:

0 No action; 1 Reclaim the directory entry and data, if unchanged.

List-Form-Duplexing Completion Code (LFDCC): The list-form-duplexingcompletion code is a value that specifies the reason a list-formduplexing process was stopped with a model-dependent timeout, when oneor more list items have been processed. It has the following encoding,as an example:

00 Timeout caused by internal conditions. 01 Command halted. 10 Implicitor explicit suppression request was accepted. 11 Duplexing is inactive.

If the command sequence number is zero, the list-form-duplexingcompletion code is set to B′00′.

Notes on the List-Form Duplexing Completion Code:

-   -   1. The LFDCC response operand provides the reason that        processing was stopped with a response code 1 at the current        list item. When the LFDCC is B′01′, a halt signal was received        during the execution of the list item and execution of the        current list item was suppressed. One of the conditions for        issuing a halt signal was encountered by the remote facility. In        this case, the duplexed command will also end with RC=1, and any        value for the LFDCC is possible. The program should retry as a        normal RC=1 condition. The LFDCC information can be used for        monitoring purposes. If the reason for halting the command        persists on the retry, then the condition will cause the first        list item to fail. In this case, the system receiving the halt        signal will most probably end with, for example, RC=18 and the        duplexed command will return the response code that identifies        the reason for halting the command.    -   2. When the LFDCC is B′10′, a request-for-suppression signal was        accepted. A possible deadlock condition has been encountered and        the duplexed command's execution of the current list item was        suppressed. In this case, the duplexed command will also end        with RC=1 and an LFDCC value of B′10′ will be returned. The        program should retry as a normal RC=1 condition. The LFDCC        information can be used for monitoring purposes.    -   3. When the LFDCC is B′11′, duplexing was broken during the        execution of the current list item. The structure state will        transition to simplex mode.    -   4. When the LFDCC is B′00′, a model-dependent timeout was        recognized by the local facility. In this case, a halt signal        was sent to the remote facility, and the duplexed command will        complete with an RC=1, and an LFDCC value of B′01′ will be        returned. It is also possible that the duplexed command        encountered the same model-dependent timeout condition, or        possibly a different halting condition and had also sent a halt        signal. In all of these cases, the duplexed command will return        an RC=1 and an LFDCC value of B′00′. These case are all normal        RC=1 conditions.

Local-Cache-Entry Deregistration Control (LCEDC): A value that controlsthe deregistration process for a Read-Directory command. It has thefollowing encoding:

0 Do not update the local-cache register; 1 Invalidate the row in eachlocal-cache register for the specified LCID.

Locked-For-Castout Selection Control (LFCSC): The locked-for-castoutselection control is a value that further controls the selection ofdirectory entries when the change-state selection control is B′1′. Thetwo possible values are:

0 Select all changed directory entries; 1 Select only directory entrieslocked for castout, where the first byte of the castout lock matches thespecified LCID.

The operand is ignored when the change-state selection control is B′0′.

Name-Block-Format Control (NBFC): The name-block-format control is a twobit value that determines the set of request operands returned in thename block by a Read-Directory command. It has the following encoding:

00 Standard name block format; 01 Return only the name field; 10 Returnthe name field and version number; 11 Invalid.

The operand is ignored unless the request type is B′10′.

Retry Index (RX): A value that designates a signaling-vector entry.Valid RXs are zero and assigned RXs within the range of one to the RXlimit. If duplexing is active and the command sequence number isnon-zero, the retry index is non-zero. If duplexing is not active forthe structure or if the command sequence number is zero, the operand isignored.

Skip-Nonexistent-Entries Control (SNEC): A value that controls thehalting of an Unlock-Castout-Locks command, when a list item specifies adirectory entry that does not exist. It has the following encoding:

0 Halt execution when the list entry does not exist; 1 Continueexecution with the next list item, when the list entry does not exist.

Storage-Class-Change Control (STCCC): A value that controls theprocessing of a reference signal when the storage class is changed. Ithas the following encoding:

0 Process the reference signal; 1 Halt execution.

Suppress Detachment Scan (SDS): A value that controls the directory scanin the Detach-Local-Cache command. It has the following encoding:

0 Scan the directory; 1 Suppress the directory scan.

Suppress Read (SR): A value that indicates the data transfer for aRead-And-Register or Read-For-Castout command is suppressed.

Suppress Registration Test (SREGT): A value that controls the testing ofthe LCEN-registration for the Write-When-Registered command. It has thefollowing encoding:

0 Test the LCEN registration; 1 Suppress testing LCEN registration.

Test-Message-Buffer-Size Indicator (TMBSI): A value that controls thetesting of the message-buffer size for the Read-And-Register command. Ithas the following encoding:

0 Do not test the message-buffer size; 1 Test the message-buffer size.

Wait-on-Ready-to-Complete Indicator (WRTCI): Thewait-on-ready-to-complete indicator is a value that determines thesignaling protocol to follow during the command execution phase of theWrite-And-Register or Write-When-Register command. It has the followingencoding:

0 Send the RTC signal as soon as execution completes. 1 Wait on sendingthe RTC signal until execution has completed and either an RTC signal orhalt- execution signal is recognized.Notes on Cache Duplexing Operands:

-   -   1. The retry buffer designated by the retry index is not written        and the contents of the retry buffer are not changed when a        cache command specifies a non-zero retry index.    -   2. Using the retry index as the addressing mechanism for the        signaling vector allows the operating system to extend its        serialization protocol for retry buffers to the signaling        vector. Since retry indices are already assigned for cache        commands, no additional serialization is needed to introduce the        signaling vector to the cache structure.    -   3. The LFCSC, LCEDC, and DTEMC operands enable the program to        perform the directory cleanup process in the Detach-Local-Cache        command explicitly. This allows the cleanup to be coordinated by        means of the duplexing command protocols, so that the state of        the duplexed directories is synchronized. The sequence used for        detaching a local cache from a duplexed pair of structures is as        follows:        -   A Read Directory command is sent to the primary cache with            the CSSC, LCEDC, and LFCSC operands set to B′1′, the request            type set to B′10′, the LCID operand set to the value of the            local-cache identifier being detached, and the            name-block-format control set to B′01′. All registrations            for the specified local cache are deregistered from the            primary cache structure.        -   The returned list of name blocks is provided as an input            list of unlock-castout-lock items on a duplexed UCL command.            The UCL command sent to the secondary cache has the IMMRC            operand set to B′1′. Both copies of the UCL command have the            DTEMC operand set to B′1′. The duplexed UCL commands are            redriven until the list is completely processed.        -   Steps 1 and 2 are repeated until the entire directory has            been scanned.        -   The final step is to independently detach the local caches            by separate Detach Local Cache (DLC) commands sent to each            cache, with the suppress detachment scan (SDS) operand set            to B′1′ for the commands sent to both structures.    -   4. The WRTCI operand should be set to B′1′ on cache write        commands sent to the secondary cache structure in the following        cases:        -   A Write-When-Registered command is duplexed with the command            sent to the primary cache having SREGT=B′0′ and the command            sent to the secondary cache having SREGT=B′1′. Setting WRTCI            in this case ensures that the registration test is performed            prior to completing the command in either structure.        -   A Write-And-Register or Write-When-Registered command is            duplexed with the command sent to the primary cache having            Version Request Type (VRT)=B′1xx′. Setting WRTCI in this            case ensures that the version number comparison is performed            prior to completing the command in either structure.        -   A Write-And-Register command is duplexed with the command            sent to the primary cache having Assignment Suppression            Control (ASC)=B′1′ and the command sent to the secondary            cache having ASC=B′0′. Setting WRTCI in this case ensures            that a directory entry is not created in the secondary cache            when the directory entry does not exist in the primary            cache.            Cache-Structure Processes for Duplexing

The following processes may be invoked by the coupling facilitycache-structure commands. The set of processes invoked by a command arelisted in the command description.

The signaling protocol for synchronizing command execution across aduplexed pair of cache structures is defined via the following set ofprocesses for duplexed command execution:

-   -   No command active process    -   Entry commit process    -   Single entry duplex command process    -   List-form duplex command process    -   Request for suppression process    -   Halting execution    -   Signal group processing    -   Execution of a duplex-signal list-notification command    -   Duplex command timeouts    -   Breaking of duplexing

These processes are described above in the section on DuplexingProcesses and are managed by the signaling protocol engine (FIG. 1). Incontrast, the description of the processes that follows is the view fromthe structure itself.

Halting a Duplexed Command Process

A single-entry or list-form duplexed command process is halted or,equivalently, completes with a halted condition, when a halt signal isrecognized, but a halt signal has not been issued. If a halt signal hasbeen issued, then the duplexed command process completes with thecondition that generated the halt signal, and any halt signal that mayhave been received is ignored.

Scanning a Directory

The directory is scanned when a Detach-Local Cache,Invalidate-Complement-Copies, Invalidate-Name, or Read-Directory commandis executed.

The directory scan is controlled by the detachment-restart token for theDetach-Local-Cache command and by the restart-token request operand forthe Invalidate-Complement-Copies, Invalidate-Name, and Read-Directorycommands. A token value of zero starts the processing, and a non-zerotoken value restarts the processing from the place designated by thetoken. Processing is completed when the entire directory has beenprocessed, when a model-dependent timeout has been exceeded, or when thecommand forces the scan to halt execution. When the end of the directoryis reached, response code 0 is returned. When a model-dependent timeoutoccurs before the end of the directory is reached, the directoryposition is generated and response code (e.g., 1) is returned. When thescan is halted, the directory position is generated and the responsecode determined by the halting condition is returned.

Note on Scanning a Directory:

-   -   1. The format of the restart token is model dependent. However,        the format depends only on the scan process and not on the        command. Thus, a portion of the directory may be scanned for        invalidation by an Invalidate-Name command and a subsequent        portion by a Read-Directory followed by an Invalidate-Name-List        command sequence, where the restart token returned by the        Invalidate-Name command is used as input to the Read-Directory        command. Likewise, a subsequent portion of the directory may be        scanned for invalidation by an invalidate-name command using the        restart token from the Read-Directory command as input. Such a        change in processing may occur when an invalidation process is        performed for a cache structure that is transforming into or out        of the duplexing-active state.        Generating a Directory Position

A directory position is a value that designates the location of adirectory entry in the cache directory. A directory position isgenerated, when one of the following commands completes with amodel-dependent timeout or is halted.

-   -   Detach local cache    -   Invalidate complement copies    -   Invalidate name    -   Invalidate name list    -   Read directory    -   Unlock castout locks.

When a Detach-Local-Cache command completes with a model-dependenttimeout, the directory position of the next directory entry to beprocessed by the directory scan is placed in the detachment-restarttoken in the local cache controls for the local cache that is beingdetached.

When an Invalidate-Complement-Copies, Invalidate-Name, or Read-Directorycommand completes with a model-dependent timeout or is halted, thedirectory position of the next directory entry to be processed by thedirectory scan is placed in the restart-token response operand.

When the Invalidate-Name-List or Unlock-Castout-Locks command completeswith a model-dependent timeout or is halted, the directory position ofthe directory entry identified by the current-list-item response operandis placed in the directory-position response operand.

When an Invalidate-Name-List or Unlock-Castout-Locks command completesprocessing and the current-list item designates a name that is notassigned to the directory and the directory position cannot bedetermined, the directory position is set to zero.

Updating a Version Number

A version number may be updated when a Write-And-Register orWrite-When-Registered command is executed, with the action takendepending on the version-request type specified, the duplexing-state ofthe structure, and the changed-state of the data.

When a version-request type of B′000′ is specified, or a version-requesttype of B′100′ is specified and version-number comparison is successful,the version number is updated as follows:

When any of the following conditions holds: (1) duplexing is not activefor the structure, (2) duplexing is active, but the command sequencenumber is zero, or (3) duplexing is active, the command sequence numberis non-zero, and the data is in the changed state, then the versionnumber is not changed. Otherwise, the version number is set to the valuezero.

When a version-request type of B′001′ is specified, or a version-requesttype of B′101′ is specified and version-number comparison is successful,the version number is updated as follows:

When any of the following conditions holds: (1) duplexing is not activefor the structure, (2) duplexing is active, but the command sequencenumber is zero, or (3) duplexing is active, the command sequence numberis non-zero, and the data is in the changed state, then the versionnumber is decremented by one. Otherwise, the version number is set tothe value minus one.

When a version-request type of B′010′ is specified, or a version-requesttype of B′110′ is specified and version-number comparison is successful,the version number is updated as follows:

When any of the following conditions holds: (1) duplexing is not activefor the structure, (2) duplexing is active, but the command sequencenumber is zero, or (3) duplexing is active, the command sequence numberis non-zero, and the data is in the changed state, then the versionnumber is incremented by one. Otherwise, the version number is set tothe value plus one.

When a version-request type of B′011′ is specified, or a version-requesttype of B′111′ is specified and version-number comparison is successful,the version-number object is set to the version-number request operand.

Note on Updating a Version Number:

When duplexing is active for the structure, unchanged data is not cachedin both structures. However, the increment and decrement functions forversion numbers rely on the presence of the version-number object in thedirectory entry as preexisting state information. If the data issubsequently written as changed or the castout lock is set, theversion-number object is to be consistent between the two structures,when the write operation is performed. Zeroing out the version numberbefore performing the increment or decrement function ensures that theduplexed pair of write commands produce the same value for the versionnumber. It may appear that this defeats the purpose of the versionnumber. However, a directory entry with no data or unchanged data may bereclaimed at any time and the reclaim operation destroys the versionnumber. A subsequent write command assigns a new directory entry andperforms the increment or decrement operation with an initial objectvalue of zero. This results in a plus one for increment and a minus onefor decrement. This is the case whether or not duplexing is active forthe structure. So, the program (i.e., the application owning thestructure, such as DB2) assumes that version numbers set in directoryentries that may be reclaimed may appear to be reset to zero at anytime. Forcing the version number to be zero when a duplexed writecommand is executed for an unchanged directory entry emulates the effectof a reclaim operation.

Suppressing Reads

The data transfer of the data area in the Read-For-Castout orRead-And-Register commands is suppressed when the suppress read (SR)request operand is set to 1. For the Read-For-Castout command, themessage-buffer size is tested, and for the Read-And-Register command,testing of the message buffer size is controlled by the TMBSI requestoperand. When the TMBSI operand is B′0′, no testing is performed. Whenthe TMBSI operand is B′1′, the message-buffer size is tested. Whentesting is performed and there is insufficient message-buffer spaceprovided, the command completes with a response code (e.g., 11).

Note on Suppressing Reads:

-   -   1. When a Read-For-Castout command is duplexed or a        Read-And-Register command which requests that data be read is        duplexed, the message buffer address list in both message        operation blocks should be set up with identical addresses and        the suppress-read operand should be set to B′0′ on the command        sent to the primary cache and should be set to B′1′. on the        command sent to the secondary cache. This ensures that        consistent checking of the message-buffer size is performed by        both coupling facilities. Testing of the message-buffer size is        performed, even when data transfer is suppressed, so that        reconciliation can be completed, when duplexing is broken during        execution of the command. Otherwise, it may be the case that the        command sent to the primary completed with response code 11, the        command sent to the secondary completed successfully, duplexing        was broken during command execution, and the secondary cache        structure was selected as the surviving structure.        Reconciliation cannot be completed in this case because the data        cannot be read into the message buffers.        Cache Command Extensions For Duplexing

The secondary cache structure differs in several ways from the primarystructure.

-   -   1. Unchanged data is not written to the secondary cache.    -   2. Registrations are not maintained in the secondary structure.        A failure that results in the secondary cache being selected to        execute in simplex mode causes the local caches to be        invalidated.    -   3. Read references are directed to the primary cache. So, the        reference order in the secondary cache is inconsistent with the        primary. However, following a failure of the primary cache, the        reference order will be restored from new references only; no        attempt is made to restore the old order.    -   4. Data-area elements are reclaimed substantially immediately        following the completion of an Unlock-Castout-Locks command,        where the change bit is zero.    -   5. Reclaim vectors are inactive in the secondary cache.

The following general changes are made to the cache commands:

-   -   1. Retry indices are added to the duplexed commands to identify        a signaling group in the signaling vector that is used to        receive signals from the remote coupling facility. However, the        corresponding retry buffers are not updated by the cache        commands.    -   2. A command sequence number (CSN) is added to the duplexed        commands to provide a time-stamp and is used to break potential        deadlocks.    -   3. Two request operands, the duplexing retry index (DRX) and the        duplexing signal group index (DSGX) are added to the duplexed        commands to construct the duplexing signals sent to the remote        facility. Additional information is provided in the duplexing        controls.

4. The current signal group index (CSGX) is returned as a responseoperand in the response descriptor for each command that exchangessignals with a remote coupling facility.

Using the above assumption for the secondary cache structure, thefollowing extensions are made to the behavior of known cache commands.This is described further below.

Allocate Cache Structure: Directed allocation, which is described inU.S. Patent Applications entitled “Method, System and Program ProductsFor Modifying Coupling Facility Structure”, Dahlen et al., Ser. No.09/379,435, filed Aug. 23, 1999; and “Directed Allocation CouplingFacility Structures, Dahlen et al., Ser. No. 09/378,861, filed Aug. 23,1999, each of which is hereby incorporated herein by reference in itsentirety, is used to create a secondary structure matching the primary,when possible. When the secondary structure is created with lessresources than the primary structure, the primary structure is alteredto match the secondary by trimming the total count objects and releasingany free segments.

Attach Local Cache: New connectors are attached in parallel to bothstructures. When a secondary cache structure is created, existingconnectors are attached as individual operations. The operating systemserializes the connect process, so command synchronization is notrequired. The values of the LCT and LCID are the same in the twostructures.

Deallocate Cache Structure: When the application requests structuredeletion, both the primary and secondary caches are deallocated, withdeallocation occurring in parallel. The operating system serializes thedeallocation process, so command synchronization is not required.Transitions from duplex mode to simplex mode causes individualDeallocate-Cache-Structure commands to be issued.

Detach Local Cache: The detach is done as a two step process. In thefirst step, a Read Directory command is issued to the primary cache toreturn all the directory entries, which are locked for castout and theLCID in the castout lock is the same as the target of the detach. A UCLcommand is then issued to both structures using the list-form protocolwith the returned list from the RD command. The second step is to issuea detach command to each structure as independent processes.

Invalidate Complement Copies: The Invalidate Complement Copies (ICC)command is only issued to the primary structure. The ICC command doesnot update any objects in the cache structure, except for thelocal-cache register and the XICIC STC counter, neither of which ismaintained in the secondary structure.

Invalidate Name (IN): The Invalidate Name command is not issued directlyusing the duplexing protocol. The Invalidate Name command scans thedirectory and deletes directory entries as they are encountered for amodel-dependent time period. If duplexed, the commands would be the twoseparate directories in an independent fashion and there is no method toensure that the directory updates are coordinated. So, it would beinevitable that the two directories would be out of synchronization forperiods of time. This is not permitted by the design and therefore, theIN command is converted into a Read-Directory command issued to theprimary followed by an Invalidate-Name-List command issued to bothstructures using the list-form duplexing protocol. On completion, therestart token generated by the Read-Directory command is returned to theissuer for use on redrive of the IN command.

Invalidate Name List: The Invalidate Name List is sent to bothstructures and the multi-command protocol is used to synchronizeexecution on a list-item basis. The command may be a direct request fromthe list-structure user, or may be the conversion of an invalidate-namerequest. When the command is a converted Invalidate Name (IN) command,the message buffer address list (MBAL) for the Send Message instructionassociated with the INL command designates the data block returned bythe Read Directory (RD) command. The formats of the list entries areidentical between the commands and do not need to be reformatted. When ahalting condition is requested, it is set on the command sent to theprimary and not set on the command sent to the secondary. The list-formprotocol halts both commands, when a proper halting event isencountered.

Process Reference List: The process reference list command is onlyissued to the primary structure. Reference ordering is not maintained inthe secondary.

Read-and-Register: Normal read-and-register requests are only sent tothe primary cache structure, since the only directory objects that areupdated are registration controls, reference bits and storage classcounters that are not maintained in the secondary cache. However, insome circumstances, the storage class, which is maintained in thesecondary cache, can be changed by the Read-And-Register (RAR) command.So, the processing for RAR is optimistically sent to the primary only,but the Read-And-Register (RAR) command is issued with a new requestoperand that tests the storage class. If the storage class would bechanged by the command, the command is completed with a new responsecode and no other action occurs. The program then reissues the RARcommand to both the primary and secondary structures using a singleentry duplexing protocol to control the execution.

Read Castout Class: Castout operations are driven off of information inthe primary structure.

Read Castout-Class Information: Castout operations are driven off ofinformation in the primary structure.

Read Directory: The reference bit does not need to be set in thesecondary, since the reference order is not maintained. A new control isadded that requests that only directory entries that are locked forcastout with a specified LCID value are returned. A new control is addedthat modifies the output of a name block, so that it is compatible withthe input list on a UCL command.

Read For Castout: Castout locks are set in both the primary andsecondary structures. The local cache entry registration control (LCERC)and name replacement control (NRC) are set to zero in the command sentto the secondary cache. The only objects updated in the secondary cacheare the castout lock, the castout count (COC), storage class (STC)counter, and the castout-class controls.

Register Name List: Since registrations are not maintained in thesecondary cache structure, the Register Name List command is only sentto the primary cache structure.

Set Reclaiming Vector: The reclaiming vector remains inactive in thesecondary cache until it is promoted to being the primary. At thispoint, the reclaiming vector may be activated. However, the contents ofthe old reclaiming vector in the primary structure has little meaning inthe secondary structure. There is a period of time where the referencesto the secondary cache structure stabilize anyway, since they do notcontain unchanged data until the structure is promoted.

Unlock Castout Lock Entry: The Unlock-Castout-Lock Entry command isissued to both structures using the single-entry duplexing protocol. Anew control is added that requests immediate reclaim, when the changebit is zero. The control is set on the command sent to the secondary.

Unlock Castout Locks: The Unlock-Castout-Locks command is issued to bothstructures using the list-form duplexing protocol. A new control isadded that requests immediate reclaim, when the change bit is zero. Thecontrol is set on the command sent to the secondary.

Write-and-Register (WAR): When the change control is one, or theoperation is write with castout, the Write-And-Register command isissued to both structures using the single-entry duplexing protocol.When the change control is zero and the operation is not write withcastout, the WAR command is only issued to the primary structure. Thecommand sent to the secondary structure, has the NRC set to zero. Wheneither the primary or secondary structure encounters atarget-storage-class-full condition, a halt signal is sent. This occursin the secondary structure when either free list is exhausted and thewrite cannot complete. Since registrations are not maintained in thesecondary, no XI signals are generated.

Write-When-Registered (WWR): When the change control is one, or theoperation is write with castout, the Write-When-Registered command isissued to both structures using the single-entry duplexing protocol.When the change control is zero and the operation is not write withcastout, the WWR command is only issued to the primary structure. A newoption is added to the WWR command to suppress the registration check.When either the primary or secondary structure encounters atarget-storage-class-full condition, a halt signal is sent. This occursin the secondary structure, when either free list is exhausted and thewrite cannot complete. Since registrations are not maintained in thesecondary, no XI signals are generated.

List-Structure Operands

Similar to cache structures, each list structure has its own operandsthat are associated with the list, when it is created. Examples of theseoperands are described below.

Comparative Structure Authority (CSAU): A value used as a comparisonvalue to the structure authority, when the structure is allocated anddeallocated, or when lock-table entries are written or list entries aredeleted and the compare-structure-authorities control is one. Thisoperand is ignored on a Delete-List or Write-Lock-Table-Entry command,when duplexing is inactive.

Comparative Remote-Facility Structure Authority (CRFSAU): A value usedas a comparison value to the remote-facility structure authority, whenlock-table entries are written or list entries are deleted and thecompare-structure-authorities control is one. This operand is ignored ona Delete-List or Write-Lock-Table-Entry command, when duplexing isinactive.

Compare-Structure-Authorities Control (CSAUC): A value that controls thecomparison of the structure authority and remote-structure authoritycontrols to the CSAU and CRFSAU operands on a Delete-List-Entries orWrite-Lock-Table-Entry command. It has the following encoding:

0 Do not compare structure authorities; 1 Compare structure authorities.

This operand is ignored unless duplexing is active for the structure.

Command Sequence Number (CSN): A value associated with a command that isduplexed. List-structure commands that specify a non-zero value in theCSN request operand cause the invocation of the duplexing-commandprocess, when duplexing is active for the structure. Commands that donot have the CSN request operand defined, or which specify a zero valuein the CSN request operand, do not invoke the duplexing-command process.

Duplexing-Active Indicator (DUPAI): A value that controls execution ofthe command based on the duplexing state of the structure. It has thefollowing encoding:

0 Do not test the duplexing state; 1 Test the duplexing state of thestructure.

Duplexing Signal-Group Index (DSGX): A value that identifies the targetsignal group in the signaling-vector entry identified by the duplexingretry index and the remote-facility controls. If duplexing is active andthe command sequence number is non-zero, the duplexing signal-groupindex is non-zero. If duplexing is not active for the structure or ifthe command sequence number is zero, the operand is ignored.

Duplexing Retry Index (DRX): A value that designates a signaling-vectorentry for the signaling vector identified by the remote-facilitycontrols. If duplexing is active and the command sequence number isnon-zero, the duplexing retry index is non-zero. If duplexing is notactive for the structure or if the command sequence number is zero, theoperand is ignored.

Failed-Structure Indicator (FSI): A value that controls the state changefor the structure that occurs when a Deallocate-List-Structure commandis executed and the structure is in the allocated state. It has thefollowing encoding:

0 Initiate deallocation of the structure; 1 Place the structure in thestructure-damage state.

EMC Restart Token (ERT): A value that determines at which EMC theRead-Event-Monitor-Control-List command restarts reading or theQueue-Pending-EMCs (QPE) command (described hereinafter) restartsscanning. Invalid values for the EMC-restart token are model dependent.

Starting List Number (SLN): A value that specifies the starting-listnumber for the Read-Event-Monitor-Controls-List or Queue-Pending-EMCscommand. The SLN is invalid, if it is greater than or equal to the listcount, or greater than the ending-list number.

Ending list number (ELN): A value that specifies the ending-list numberfor the Read-Event-Monitor-Controls-List (REMCL) or Queue-Pending-EMCs(QPE) command. For the REMCL command, the ELN is invalid, if it isgreater than or equal to the list count, or less than the starting-listnumber. For the QPE command, any value for the ELN operand is valid.

Intermediate-Controls-Returned-on-Timeout Control (ICRTOC): A value thatcontrols the completion processing of a Delete-List-Entries command,when a model-dependent timeout is recognized. The two possible valuesare:

0 List-entry controls are not returned on a timeout condition. 1List-entry controls for an intermediate list entry in the scan arereturned on a timeout condition.

This operand is ignored unless the skip-nonexistent entries control(SNEC) (described below) and list number comparison type (LNCT) operandsare both B′1′.

List-Form-Duplexing Completion Code (LFDCC): The list form-duplexingcompletion code is a value that specifies the reason a list-formduplexing process was stopped with a model-dependent timeout, when oneor more list items have been processed. It has the following encoding:

00 Timeout caused by internal conditions. 01 Command halted. 10 Implicitor explicit suppression request was accepted. 11 Duplexing is inactive.

If the command sequence number is zero, the list-form-duplexingcompletion code is set to B′00′.

Notes on LFDCC Response Operand:

-   -   1. The LFDCC response operand provides the reason that        processing was stopped with a response code 1 at the current        list item.    -   2. When the LFDCC is B′01′, a halt signal was received during        the execution of the list item and execution of the current list        item was suppressed. One of the conditions for issuing a halt        signal was encountered by the remote facility. In this case, the        duplexed command will also end with RC=1, and any value for the        LFDCC is possible. The program should retry as a normal RC=1        condition. The LFDCC information can be used for monitoring        purposes. If the reason for halting the command persists on the        retry, then the condition will cause the first list item to        fail. In this case, the system receiving the halt signal will        most probably end with RC=18 and the duplexed command will        return the response code that identifies the reason for halting        the command.    -   3. When the LFDCC is B′10′, a request-for-suppression signal was        accepted. A possible deadlock condition has been encountered and        the duplexed command's execution of the current list item was        suppressed. In this case, the duplexed command will also end        with RC=1 and an LFDCC value of B′10′ will be returned. The        program should retry as a normal RC=1 condition. The LFDCC        information can be used for monitoring purposes.    -   4. When the LFDCC is B′11′, duplexing was broken during the        execution of the current list item. The structure state will        transition to simplex mode.    -   5. When the LFDCC is B′00′, a model-dependent timeout was        recognized by the local facility. In this case, a halt signal        was sent to the remote facility, and the duplexed command will        complete with an RC=1, and an LFDCC value of B′01′ will be        returned. It is also possible that the duplexed command        encountered the same model-dependent timeout condition, or        possibly a different halting condition and had also sent a halt        signal. In all of these cases, the duplexed command will return        an RC=1 and an LFDCC value of B′00′. These case are all normal        RC=1 conditions.

List-Set Position (LSP): A value that denotes a position of a list entryin the list set.

Read-LZIDs Indicator (RLEIDI): A value that indicates whether the datablock contains a list of LEIDs or contains the information specified bythe RLT request operand. It has the following encoding:

0 Return information specified by the RLT operand; 1 Return a list ofLEIDs only.

Retry Index (RX): A value that designates a retry buffer and asignaling-vector entry. An RX of zero indicates that the retry buffershould not be written. If the retry index is non-zero and the command isamong the list of commands in the process for writing the retry buffer,the retry buffer is written. Valid RXs are zero and assigned RXs withinthe range of one to the RX limit. If duplexing is active and the commandsequence number is non-zero, the retry index is non-zero.

Skip-Nonexistent-Entries Control (SNEC): A value that controls thehalting of a Delete-List-Entries command, when a list item specifies alist entry that does not exist. It has the following encoding:

0 Halt execution when the list entry does not exist; 1 Continueexecution with the next list item when the list entry does not exist.

Suppress-Notification Control (SNC): A value that controls the sendingof list notification commands when a list-state transition, key-rangetransition, or event queue state transition occurs, and controls boththe queuing and withdrawing of EMCs, when a subsidiary-list statetransition occurs. It has the following encoding:

0 Queue or withdraw EMCs and issue list-notification commands; 1 Do notqueue or withdraw EMCs or issue list- notification commands.

Suppress Read (SR): A value that indicates the data transfer for a readcommand is suppressed, but the data-list entry is written to the retrybuffer.

List-Structure Processes for Duplexing

The following processes may be invoked by the list-structure commands.The set of processes invoked by a command are listed in the commanddescription. The signaling protocol for synchronizing command executionacross a duplexed pair of list structures is defined via the followingset of processes for duplexed command execution:

-   -   No command active process    -   Entry commit process    -   Single entry duplex command process    -   List-form duplex command process    -   Request for suppression process    -   Halting execution    -   Signal group processing    -   Execution of a duplex-signal list-notification command    -   Duplex command timeouts    -   Breaking of duplexing.

These processes are described in the section on duplexing signals andare managed by the signaling protocol engine (FIG. 1). In contrast, thedescription of the processes that follows is the view from the structureitself.

Writing the Retry Buffer: The following commands update the contents ofthe specified retry buffer when the retry buffer is assigned:

-   -   Clear lock table    -   Delete list    -   Delete list entries    -   Delete list entry    -   Delete list set    -   Dequeue event-monitor controls    -   Move and read list entry    -   Move list entries    -   Move list entry    -   Perform adjunct lock operation    -   Perform adjunct lock operations    -   Read and delete list entry    -   Record global lock manager    -   Withdraw adjunct lock user    -   Write and move list entry    -   Write list controls    -   Write list entry    -   Write lock-table entry.

When the retry index is zero, no retry buffer is updated. When the retryindex is non-zero, the retry-version-number request operand and theresponse operands, except for the response descriptor, are stored in theretry-information portion of the retry buffer specified by the retryindex. When the retry index is non-zero and a data list entry is read,or when the list-entry type specifies reading the data list entry, butthe suppress-read operand is b′1′, the data list entry is also stored inthe retry-data-block portion of the retry buffer specified by the retryindex.

The duplexing-deactivated indicator is copied from bit 28 of word 2 inthe response descriptor to bit 0 of word 1 of the retry information.

When the command is terminated, suppressed, or completed such that thecompletion appears the same as suppression except that an MRB may bereturned, the retry buffer may or may not be updated.

Halting a Duplexed Command Process: A single-entry or list-form duplexedcommand process is halted, or, equivalently, completes with a haltedcondition, when a halt signal is recognized, but a halt signal has notbeen issued. If a halt signal has been issued, then the duplexed commandprocess completes with the condition that generated the halt signal, andany halt signal that may have been received is ignored.

Scanning a List Set: The list set is scanned when a Delete-List-Set orRead-List-Set command is executed. The list-set scan is controlled bythe restart-token request operand. A token value of zero starts theprocessing, and a non-zero token value restarts the processing from theplace designated by the token. Processing is completed when the entirelist-set has been processed, when a model-dependent timeout has beenexceeded, or when the command forces the scan to halt execution. Whenthe end of the list set is reached, response code 0 is returned. When amodel-dependent timeout occurs before the end of the directory isreached, the list-set position is generated and response code 1 isreturned. When the scan is halted, the list-set position is generatedand the response code determined by the halting condition is returned.

Generating a List-Set Position: A list-set position is a value thatdesignates the location of a list entry in the list set. A list-setposition is generated, when one of the following commands completes witha model-dependent timeout or is halted:

-   -   Delete List Entries;    -   Delete List Set;    -   Read List Set.

When a Delete-List-Set or Read-List-Set command completes with amodel-dependent timeout or is halted, the list-set position of the nextlist entry to be processed by the list-set scan is placed in therestart-token response operand.

When the Delete-List-Entries command completes with a model-dependenttimeout or is halted, the list-set position of the list entry identifiedby the current-data-index response operand is placed in thelist-set-position response operand.

When a Delete-List-Entries command completes processing and thecurrent-data index designates a list entry that does not exist and thelist-set position cannot be determined, the list-set position is set tozero.

Suppressing Reads

The data transfer of the data area in the Read-List-Entry,Move-And-Read-List-Entry, Read-And-Delete-List-Entry commands issuppressed, when the SR request operand is set to 1. However, themessage-buffer size is still tested to see if sufficient message bufferspace is provided for returning the data area. If there is insufficientmessage-buffer space provided, the command completes with a responsecode (e.g., 11). Additionally, the data-list entry is moved to theretry-data-block portion of the retry buffer specified by the retryindex.

Notes on Suppressing Reads:

-   -   1. When a Read-List-Entry, Read-And-Delete-List-Entry, or        Move-And-Read-List entry command is duplexed, the message buffer        address list in both message operation blocks should be set up        with identical addresses and the suppress-read operand should be        set to B′0′ on the command sent to the primary list structure        and should be set to B′1′ on the command sent to the secondary        list structure. This ensures that consistent checking of the        message-buffer size is performed by both coupling facilities.        Testing of the message-buffer size is performed even when data        transfer is suppressed, so that reconciliation can be completed        when duplexing is broken during execution of the command.        Otherwise, it may be the case that the command sent to the        primary completed with response code 11, the command sent to the        secondary completed successfully, duplexing was broken during        command execution, and the secondary list structure is selected        as the surviving structure. However, reconciliation cannot be        completed because the data cannot be read from the retry buffer        into the message buffers.        List Command Extensions for Duplexing

In one embodiment, the primary and secondary list structure are kept insynchronization with the exception of the event queues. The secondarylist appears to be a duplicate of the primary list (except for eventqueues). This requires that virtually every command be duplexed andsynchronized, including some read commands.

The event queues are only maintained, in this example, in the primarystructure. However, the key structures in the secondary list include allthe state information employed to restore the event queues on failover.This is done by the LFSS issuing a QPE command (described herein) to thesecondary during the failover process.

The following general changes are made to the list commands:

-   -   1. There is one list-notification vector per connector. List        notifications are not generated in the secondary list. A new        suppress-notification control is added to commands that generate        LNs.    -   2. Retry indices are added to the duplexed commands to identify        a signaling group in the signaling vector that is used to        receive signals from the remote coupling facility. However, the        corresponding retry buffers are only updated for the commands        that previously updated the retry buffer. In particular, the        locking commands do not update the retry buffer.    -   3. A command sequence number (CSN) is added to the duplexed        commands to provide a time-stamp and is used to break potential        deadlocks.    -   4. Two request operands, the duplexing retry index (DRX) and the        duplexing signal group index (DSGX), are added to the duplexed        commands to construct the duplexing signals sent to the remote        facility. Additional information is provided in the duplexing        controls.    -   5. The current signal group index (CSGX) is returned as a        response operand in the response descriptor for each command        that exchanges signals with a remote coupling facility.

Allocate List Structure: Directed allocation, which is described in U.S.Patent Applications entitled “Method, System and Program Products ForModifying Coupling Facility Structure”, Dahlen et al., Ser. No.09/379,435, filed Aug. 23, 1999; and “Directed Allocation CouplingFacility Structures, Dahlen et al., Ser. No. 09/378,861, filed Aug. 23,1999, each of which is hereby incorporate herein by reference in itsentirety, is used to create a secondary structure matching the primary,when possible. When the secondary structure is created with lessresources than the primary structure, the primary structure is alteredto match the secondary by trimming the total count objects and releasingany free segments.

Attach List-Structure User: New connectors are attached in parallel toboth structures. When a secondary list structure is created, existingconnectors are attached as individual operations. The operating systemserializes the connect process, so command synchronization is notrequired. The values of the LNT and UID are the same in the twostructures.

Cleanup Lock Table Entries: A disconnect causesCleanup-Lock-Table-Entries commands to be issued in parallel to bothstructures. The operating system serializes the disconnect process, so aduplexing protocol to ensure synchronization is not required. Issuingthe cleanup separately implies that the resetting of the lock-tableentries is not synchronized between the structures. This is acceptable,since the user connection has been invalidated, and thus, no new lockcommands will be processed for this UID.

The following commands are issued to both structures using thesingle-entry duplexing protocol to control their execution:

Clear Global Lock Manager

Delete List Entry

Deregister List Monitor

Move and Read List Entry

Move List Entry

Read and Delete List Entry

Read List Entry

Record Global Lock Manager

Register List Monitor

Reset Global-Lock Manager

Set Global-Lock Manager

Set Local-Lock Manager

Write and Move List Entry

Write List Controls

Write List Entry

Write Lock-Table Entry.

The following commands are issued to both structures using the list-formduplexing protocol to control their execution:

Delete List Entries

Dequeue Event Monitor Controls List

Move List Entries

Register Event Monitors

Reset Lock Managers.

The following commands are issued to the primary structure only:

Read Event Monitor Controls

Read EMC List

Read Event-Queue Controls

Read List

Read List Controls

Read List Set

Read Lock-Table Entry

Read Lock-Table Range

Read Next Lock-Table Entry

Write List-Set Scan Controls.

Clear Lock Table: This command is used when the last connectordisconnects and the structure is persistent. Since all activity to thestructure has ceased, the command can be issued independently to the twostructures, without duplexing controls.

Dealocate List Structure: When the application requests structuredeletion, both the primary and secondary lists are deallocated, withdeallocation occurring in parallel. The operating system serializes thedeallocation process, so command synchronization is not required.Transitions from duplex mode to simplex mode will cause individualDeallocate-List-Structure commands to be issued.

Delete List: Delete list (DL) is converted into a Read List commandissued to the primary followed by a Delete-List-Entries command issuedto both structures with the list-form duplexing protocol. On completion,the restart token generated by the Read List command is returned to theissuer for use on redrive of the DL command.

Delete List Entries: The command may be a direct list-structure userrequest or may be the conversion of a delete-list-set request. A controlsuppresses LNs in the secondary. The format of the data block matchesthe format for the Read-List-Set command.

Delete List Set: The DLS command is converted into a Read-List-Setcommand issued to the primary followed by a Delete-List-Entries commandissued to both structures with the list-form duplexing protocol. Oncompletion, the restart token generated by the Read List Set command isreturned to the issuer for use on redrive of the DLS command.

Dequeue Event Monitor Controls (DEMC): The DEMC command is convertedinto a Read-EMC-List command issued to the primary followed by aDequeue-EMC-List command issued to both structures with the list-formduplexing protocol.

Dequeue Event Monitor Controls List: A list-form of the DEMC command isused to synchronize the dequeue operations. The data-block formatmatches the format for the Read-EMC-List command.

Queue Pending EMCs (QPE):

One example of the request operands provided in the message commandblock for the QPE command are summarized in the following table.

Request Operands Acronym Message Header Command Code CC StructureIdentifier SID EMC Restart Token ERT Starting List Number SLN EndingList Number ELN

In execution of one embodiment of the QPE command, if the value of theevent-monitor-controls-count object is zero, the command is completedand a response code (e.g., 0) is returned. Otherwise, the event monitorcontrols within the list set are scanned starting with the starting-listnumber or the EMC-restart token, until a model dependent time periodelapses or the last event monitor control is scanned. A zero EMC-restarttoken causes the entire list to be processed starting at the start-listnumber operand. A valid non-zero EMC-restart token starts the processingat the event monitor control object designated by the EMC-restart token.

The EMCs are scanned starting with the starting-list number, then inascending order by LN up to either the ending-list number or to one lessthan the list count, whichever is smaller. The EMCs in a list-number arescanned in an unpredictable ordering for keys, and an unpredictableordering for UIDs within a key value.

The event-monitor controls in the list set are processed by queueingeach EMC that is queue-pending to the corresponding event queue and bywithdrawing each EMC that is withdrawal pending from the correspondingevent queue. If this causes event-queue transitions, the event-queuemonitors are notified. The queueing or withdrawing of event-monitorcontrols and the generated list-notification commands are primaryprocesses.

The list-set scan ensures that any EMC that is queue-pending orwithdraw-pending when the scan is initiated and remains queue-pending orwithdraw-pending throughout the scan is queued to or withdrawn from theevent queue, as appropriate.

When a model-dependent time period has elapsed, the list-set positionfor the next EMC to be processed is generated and placed in theEMC-restart-token response operand. The EMC-restart token and a responsecode (e.g., 1) are returned.

When the list-set scan is completed, a response code (e.g., 0) isreturned.

When the EMC-restart token is invalid, a response code (e.g., 3) isreturned.

The following response codes may be returned:

List-scan completed;

Model-dependent timeout;

Invalid EMC-restart token.

Detach List-Structure User: A disconnect causesDetach-List-Structure-User commands to be issued in parallel to bothstructures. The operating system serializes the disconnect process, socommand synchronization is not required. Issuing the detach separatelyimplies that the dequeuing of the EMCs is not synchronized between thestructures. This is acceptable, since the user connection has beeninvalidated, and thus, no new EMCs will be queued for this UID.

Read List-Structure Controls: Depending on how structure information isreported, the information may only be obtained from the primarystructure, or obtained in an independent fashion.

Read User Controls: While the LNT, US and SYID are the same between thestructures, the User Authority (UAU) and user attachment control (UAC)may be different.

Described in detail above is a capability that allows couplingfacilities to be coupled to one another via, for instance, a peer link.The coupling of the facilities allows many functions to be employed,including the duplexing of structures. Although duplexing is describedabove, the coupling of the facilities can be for reasons other thanduplexing.

The duplexing of structures results in two structures being created intwo different coupling facilities. In one example, the couplingfacilities are failure isolated, so that the failure of one does notaffect the other.

While both cache and list (including lock) structures can be duplexed,the information that is duplexed is different for the different types ofstructures, as described herein. In other embodiments, however, otherinformation may or may not be duplexed.

Although in the embodiments described herein, the duplexing results intwo structures of two coupling facilities, this can be extended to aplurality of structures in a plurality of coupling facilities.

In the embodiments described above, various objects controls andoperands are described. These are only examples. There may be more, lessor different objects, controls and/or operands. Further, in someexamples, the values of a bit or response code may be provided. Again,these values are only examples. Any other values may be used. Moreover,in the various control flows, various tests are performed. These areonly examples. Tests may be added or deleted, depending on the Sysplex.For example, if dumping serialization is not a part of the Sysplex, thenthe tests associated with dumping serialization can be eliminated. Thesame is true for other tests.

Many advantages are provided by the various aspects of the presentinvention. Examples of these advantages are described below:

-   -   No new hardware changes are required. The peer link between the        two coupling facilities is the same physical link that is used        for connecting host systems to coupling facilities. In fact, any        of the three coupling link types, the intersystem channel (ISC),        the integrated cluster bus (ICB), or the internal coupling link        (IC), can be used for the purpose of exchanging signals between        the coupling facilities for duplexing. Moreover, when one or        both coupling facilities are internal coupling facilities        (ICFs), the same links can be used for both duplexing traffic        and for normal command traffic.    -   No new link architecture is required. The signaling protocol        used for duplexing utilizes the list-notification mechanism that        already exists in the coupling link architecture. The duplexing        signals are defined as unique encodings of the        list-notification-entry number and address a list vector that is        created by the coupling facility and accessed by a standard        list-notification token that is exchanged between the coupling        facilities.    -   The peer link design is highly scalable. In particular, the data        rates for signal exchanges are very low compared to the data        rates for commands and data exchanged between host systems and        the coupling facilities. So, a single peer link can support the        combined signaling traffic generated by all the coupling links        attached from the host systems to the coupling facilities for        all the structures that are duplexed.    -   The peer link design is highly available. Multiple peer links        can be configured as redundant coupling facility-to-coupling        facility connections and the duplexing protocol will recognize        link failures and maintain the signal exchange on surviving        links.    -   The peer link design is highly flexible. Coupling        facility-to-coupling facility links need not be configured        between all pairs of coupling facilities, only the ones in which        duplexed pairs of structures are to reside. A coupling facility        may be connected to many other coupling facilities in this way,        and duplexed structures may be located in any of the connected        coupling facilities.    -   Duplexed coupling facility operations, including data transfer,        can be performed in parallel to the coupling facilities. This        contrasts with the store-and-forward design, or alternatively, a        design that sends a command first to one coupling facility and        then, after the first completes, to a second coupling facility.        The net result of parallel execution is that the elapsed        execution time for a duplexed pair of operations should        approximate the elapsed time of a single command.    -   Signaling between the coupling facilities is nondisruptive. The        receiver channel makes storage updates to reflect receipt of        coupling facility-to-coupling facility signals, without needing        to interrupt or otherwise, get the attention of the coupling        facility code at that time.    -   The granularity of duplexing is on a structure basis. No fixed        association exists between duplexed structures and coupling        facilities. For instance, three coupling facilities may be used        to duplex two separate structures, where only one of the        coupling facilities contains both structures. Also, some        structures in a given coupling facility may be in the duplexed        state and others may be simplexed (no duplexed copy). Also,        moving between states, duplexed-to-simplexed or        simplexed-to-duplexed, is done on an individual structure basis.    -   Existing CFCC latching mechanisms serialize command execution        and command atomicity properties with only the addition of the        duplexing signals to coordinate the execution of the commands.        No external serialization is required to serialize the execution        of the commands. That is, locking structures outside of the        structure itself are not needed nor is any serialization needed        by those other than the coupling facility. Further, it is not        necessary to try to simulate or reproduce the coupling        facilities existing internal atomicity properties via some new        external serialization protocol. The existing latching        mechanism, and all that it implies, is preserved intact.    -   The LFSS component is extended to obtain two subchannels in        parallel when a command is split for duplexing and to        transparently (to the requesting application) handle all aspects        of executing the two commands, as if they were a single command,        including: handling error conditions, retries, and merging        results.    -   An extension to the message-facility architecture, called the        synchronous completion on initial status (SCIS) bit, is defined        that allows for optimal redrive of one or both duplexed commands        in the presence of link congestion. This function is used for        existing simplex requests, as well as the duplexed requests, and        is an advantage in both environments. However, in the duplexing        environment it not only improves command elapsed time, but also        minimizes command skew when one of the two commands encounters        congested links and the second does not.

In addition to the above advantages, the following is provided as asummary of various aspects of the present invention.

In order to duplex coupling facility structures and commands againstthose structures, an efficient means for the two coupling facilitiesparticipating in duplexing is provided to synchronize their commandprocessing for a particular duplexed operation against a particularduplexed structure. Duplexed coupling facility commands execute inharmony between the two coupling facilities in a way which preservesproperties of command atomicity/concurrency guaranteed by the couplingfacility command architecture for simplex structures. Both commandseither complete, or back out, in the two coupling facilities. In thiscontext, a highly-efficient means of signaling between the two couplingfacilities to communicate status of processing of the request isemployed.

In one example, the mechanism is based on an application of the existingList Notification (LN) mechanism that coupling facilities use today tocommunicate status information to the attached operating systems. Herethis mechanism is used for coupling facility-to-coupling facilitycommunication.

An architected encoding of the list notification entry number (LNEN) foruse in coupling facility-to-coupling facility signaling provides fivesignal types:

RTC (Ready Used to extend command concurrency rules to Complete) for asingle coupling facility command to a Signal duplexed pair of commands.In particular, this signal indicates that the coupling facilities areready to complete command processing and commit the results for aduplexed operation. RTE (Ready Used to minimize latch hold times for toExecute) skewed commands by delaying latch obtains Signal until bothcoupling facilities have received the MCBs. Halt Signal Allows fordifferent resource usage patterns in the 2 coupling facilities. Couplingfacilities do not need to be duplexed in their entirety, rather they areduplexed (or not) on a structure basis. Coupling facilities can bedifferent functional LEVELs and/or different implementations. It alsoallows the duplexing of changed data only for cache structures. (This isa very flexible signal and has solved a number of other problems aswell - link timeout skews, deadlock avoidance without RTC reception, forinstance.) RFS (Request Along with the CSN operand, these signals forprovide a deadlock avoidance mechanism Suppression) where requestsrequiring the same coupling and RFSA facility resources in order toexecute, (Request for arrive and begin execution in reversed Suppressionorder in the two coupling facilities Accepted) participating induplexing. Using these Signals signals, one of the commands releases itsresources and is “suppressed”, allowing the other to executesuccessfully. The operating system will then redrive the suppressedcommand.

As described herein, in one embodiment, the use of three entries pervector index in a round-robin pattern (+CSGX cursor) reduces theserialization and recovery design of lost or delayed signals.

Use of the retry index provides for an automatic mechanism for assigningsignaling vector entries. A second serialization protocol among theSysplex members is avoided.

Use of existing message facility mechanisms simplifies the design anddoes not require a new storage allocation process for the vectors.

In a further aspect, “read secondary” processing is provided, whichallows the software to retrieve the data on a read command from thesecondary structure after an IFCC results in the data not being readfrom the primary structure. Even though the read of the data wassuppressed on the duplexed read command to the secondary, the data wasstaged into a retry buffer so that it could be retrieved in the event ofsuch a failure.

A new global coupling facility object, the Duplexing Active vector, isdefined. It is a parallel structure to the SID vector, and is likewiseindexed by the Structure Identifier (SID) operand. Each bit in theDuplexing Active vector corresponds to the current duplexing state ofthe corresponding structure; the state of the bit therefore, serves tocontrol command execution in duplex versus simplex mode for eachstructure in the coupling facility (of course, at a finer level ofgranularity, individual coupling facility commands executed against aduplexed structure may be executed either as simplex commands orduplexed commands).

This structure provides a number of functions employed by the duplexingarchitecture model and it provides a number of distinct advantages forthe overall duplexing framework:

-   -   The granularity of duplexing is on a structure basis. Each        coupling facility may have a mixture of structures in simplex        state and duplex state (and, when duplexed, the various duplexed        peer structures may reside in a mixture of different coupling        facilities, which are all connected to this coupling facility).        The duplexed entities are structures, not coupling facilities.        This provides the customer with considerable flexibility in        configuring the Parallel Sysplex environment to meet        availability objectives for critical structures, workloads and        applications, while lowering the overall cost and complexity of        the configuration by allowing other non-critical structures to        continue to execute in simplex mode. This flexibility also        enhances the customers' ability to test the coupling facility        duplexing function in a limited way prior to rolling it out        broadly throughout the installation, and greatly simplifies the        migration path from today's simplex environment to an        environment where duplexing is being used extensively.    -   The duplexing active bit for a structure may only be set by the        operating system via the Activate Duplexing (ADPLX) command.        This provides strict OS control over entering the duplexed        state. (In other embodiments, this control may be eased.)    -   When duplexing is activated by the OS, structure-related        duplexing controls uniquely identify a duplexed structure's peer        structure instance (RFSID, RFSAU) and the coupling facility        instance in which it resides (RFND, RFSYID). This allows tight        control over which structures represent valid duplex copies of        which other structures, and ensures that duplexing signals are        sent to the correct coupling facility instance which contains        this peer structure. This same information can also be used by        the OS after sysplex-wide failures or total loss of connectivity        to a coupling facility, to ensure that the duplexed state is        preserved for valid pairs of duplexed structures, whenever        possible.    -   The duplexing active bit for a structure may be reset by either        the OS (via the Deactivate Duplexing (DDPLX) command) or by the        CFCC. Allowing the OS to reset the bit provides for        configuration control in the OS. Allowing the CFCC to reset the        bit ensures that a failure detected by the coupling facility        results in a consistent state for all of the duplexed objects in        the coupling facilities, since once the bit is reset for a        structure, no subsequent execution of commands in a duplexed        fashion occurs. Duplexed command execution processes only        operate, when the bit in the Duplexing Active vector is set for        a structure. This simplifies failover, since the recovery is        limited to those commands that were executing at the time the        failure was detected (not any subsequent commands), and the        resulting state of the objects affected by the commands is well        defined.    -   Duplexed commands issued after duplexing is broken cause        immediate command suspension with (e.g., RC=20) that allows        systems to detect the change in state without updating coupling        facility objects and without relying on a message exchange in        the sysplex. This leaves the coupling facility structure in a        consistent state until recovery/failover can be coordinated.    -   The technique to break duplexing (and reset the duplexing active        bit) in the coupling facility is extremely tolerant of temporary        loss of the coupling facility-to-coupling facility connection,        with the advantage that the duplexed state of the structures is        far more robust and highly available than it might have been        otherwise. If a single link between the coupling facilities        fails, and other redundant links connecting the pair of coupling        facilities exist, then those other links will be used to send        the signals, and duplexing will be preserved for the structures.        If all links between coupling facilities fail, and then one or        more of the links is recovered and no duplexing commands have        been processed while all links were not operational, then        duplexing will be preserved for the structures. Furthermore,        even if duplexing commands are processed while all links between        the coupling facilities are in a not-operational state, the        signaling protocol tolerates this by using an initial Ready To        Execute signal (RTE), which by itself does not permit structure        objects to be updated; in the event that the loss of coupling        facility-to-coupling facility connectivity only prevents these        initial signals from being exchanged, the coupling facility need        not break duplexing. Rather, it will report a “path not        available” condition to software, who will then use a Test        Remote Facility Access (TRFA) command to patiently wait for the        coupling facility-to-coupling facility link to be restored and        hold the command in abeyance, for a time. If coupling        facility-to-coupling facility connectivity is restored within        the timeout period, then duplexing is preserved and the duplexed        commands that experienced the “path not available” condition are        redriven; if not, then duplexing is broken by the software.    -   When the coupling facility connection fails, duplexing is only        broken for those structures with images in each coupling        facility. Also, the detection of the state change is done when        the next command is executing for the structure. This ensures        that the correct OS images see the error (i.e., no other        reporting mechanism for reporting “duplexing broken” is needed),        and see it at a time when recovery/failover can occur.

In further aspects of the invention, other architectural extensions andprocessing optimizations have been added to the coupling facilitycommand architecture in support of coupling facility duplexing. Many ofthese are intrinsic to the basic single-entry and list-form duplexingprocesses themselves, in order to enable the duplexed structure objectsto be updated consistently and maintained in a synchronized stateindefinitely. Many others enhance the support by contributing to therobustness, transparency, manageability, and performance/efficiency ofthe coupling facility duplexing processes and protocols.

The following areas are extended, as examples:

-   -   1. Secondary structure allocation and copy processing support.        New commands support the determination of coupling        facility-to-coupling facility peer link connectivity, used by        the OS to constrain allocation of “duplex capable” structures,        and to give preference to coupling facilities that are        failure-isolated from one another, etc. Furthermore, the use of        directed allocation is extended to ensure that the secondary        structure is allocated as a duplicate of the primary, in terms        of number of structure objects. This enhances transparency as        viewed by programs using the structure, and facilitates the        ability to failover to either the primary or secondary structure        and then operate in simplex mode, when necessitated by a failure        condition affecting one of the structure instances.    -   2. “Double” commands and duplexing. Suppression operands        (suppress notification, suppress registration, suppress read)        allow existing commands to be duplexed without any explicit        execution modes defined in the coupling facility (no explicit        primary mode or secondary mode for a given structure instance).    -   3. “Triple” commands (also called “converted” commands) allow        for transparent duplexing of commands that cannot be duplexed        with either the single-entry or list-form duplex protocol,        because signals cannot synchronize the execution of the        “structure scan” processes these commands invoke (an example of        such a command is Delete List Set). These are converted to        “triple” commands: The first of the three reads a list of        entries to be processed, from the primary structure; the second        and third of the three execute a duplexed command using the        list-form duplex protocol, using the list returned by the first        of the three commands.    -   4. Common/interchangeable restart tokens across the command set,        along with the common DIRP/LSP operands to denote directory        position or list set position in commands which scan through the        structure. This interchangeability enables the use of commands        other than those actually requested by the connector, which is        intrinsic to the “triple command” architecture and mechanism.    -   5. Response code reconciliation provides a single, consistent        response to the exploiter.    -   6. Detach emulation processing for cache structures, and lock        table cleanup processing for lock structures. These allow        cleanup of structure objects to be performed consistently        between the duplexed structures, despite the “structure scan”        processes they involve and the inherent difficulty in        synchronizing such scans across the structures.    -   7. Optimizations for cache structures by duplexing only changed        or locked-for-castout entries; unchanged entries and data are        not duplexed.    -   8. Copy process optimizations (do not copy cache registrations        nor unchanged entries/data when establishing the secondary copy        of the structure initially).    -   9. Mainline command request optimizations (Writes of unchanged        data not locked for castout are written only to primary. RAR and        RNL are normally driven only to primary, unless a change in        storage class is processed.    -   10. Optimization for “pure reads” that do not modify structure        objects being driven in simplex mode to the primary structure        only, or from the “closer” coupling facility, whichever one it        may be (primary or secondary).    -   11. Optimization for event queue monitoring to not duplex the        event queues; allows the event queue monitoring queuing and        withdrawal processes to continue to be performed as secondary        processes asynchronous to the completion of the originating        command. Requires additional processing when failing over to the        secondary structure.    -   12. Optimization to IN/INL commands to allow them to be        processed simplex to the primary structure when the invalidation        type indicates processing unchanged data.    -   13. Coordination of IFCC recovery.    -   14. Immediate reporting of busy conditions minimizes skew in the        duplexed pair (SCIS bit support). Further, the ability to send        the commands in parallel to the coupling facilities, so the        elapsed time for the duplexed pair should approximate the        elapsed time of a single command.    -   15. Performance/measurement counters.

The present invention can be included in an article of manufacture(e.g., one or more computer program products) having, for instance,computer usable media. The media has embodied therein, for instance,computer readable program code means for providing and facilitating thecapabilities of the present invention. The article of manufacture can beincluded as a part of a computer system or sold separately.

Additionally, at least one program storage device readable by a machine,tangibly embodying at least one program of instructions executable bythe machine to perform the capabilities of the present invention can beprovided.

The flow diagrams depicted herein are just examples. There may be manyvariations to these diagrams or the steps (or operations) describedtherein without departing from the spirit of the invention. Forinstance, the steps may be performed in a differing order, or steps maybe added, deleted or modified. All of these variations are considered apart of the claimed invention.

Although preferred embodiments have been depicted and described indetail herein, it will be apparent to those skilled in the relevant artthat various modifications, additions, substitutions and the like can bemade without departing from the spirit of the invention and these aretherefore considered to be within the scope of the invention as definedin the following claims.

1. A method of coupling a plurality of coupling facilities of acomputing environment, said method comprising: selecting a firstcoupling facility and a second coupling facility to be coupled, whereinat least one coupling facility of said first coupling facility and saidsecond coupling facility comprise shared storage that includes activestate information shareable by one or more distributed applications ofthe computing environment; and providing at least one peer connectionbetween the first coupling facility and the second coupling facility tocouple the first coupling facility and the second coupling facility. 2.The method of claim 1, wherein a peer connection of the at least onepeer connection comprises an intersystem channel.
 3. The method of claim1, wherein a peer connection of the at least one peer connectioncomprises an integrated cluster bus.
 4. The method of claim 1, wherein apeer connection of the at least one peer connection comprises aninternal coupling link.
 5. The method of claim 1, wherein said at leastone peer connection comprises a plurality of peer connections, andwherein multiple peer connections of the plurality of peer connectionsare configured redundant connections between the first coupling facilityand the second coupling facility.
 6. The method of claim 1, wherein eachof the first coupling facility and the second coupling facility is aspecial purpose storage processor.
 7. The method of claim 1, whereineach of the first coupling facility and the second coupling facilityexecutes code other than user application code.
 8. The method of claim1, wherein a peer connection comprises a protocol for command initiationthat is capable of being initiated from the first coupling facility andthe second coupling facility.
 9. The method of claim 5, furthercomprising: recognizing a failure of a peer connection of the multiplepeer connections; and maintaining an exchange of information on one ormore surviving peer connections of the multiple peer connections.
 10. Amethod of coupling a plurality of coupling facilities of a computingenvironment, said method comprising: selecting a first coupling facilityand a second coupling facility to be coupled; and providing at least onepeer connection between the first coupling facility and the secondcoupling facility to couple the first coupling facility and the secondcoupling facility, wherein the first coupling facility and the secondcoupling facility are coupled to provide duplexing of one or morecoupling facility structures.
 11. The method of claim 10, wherein aninstance of one coupling facility structure of said one or more couplingfacility structures exists in said first coupling facility and saidsecond coupling facility, providing a first coupling facility structureinstance and a second coupling facility structure instance.
 12. Themethod of claim 11, further comprising maintaining synchronization ofexecution of commands issued against said first coupling facilitystructure instance and said second coupling facility structure instance.13. The method of claim 12, wherein said maintaining synchronizationcomprises exchanging one or more signals across one or more peerconnections of said at least one peer connection.
 14. The method ofclaim 12, wherein commands are issued against said first couplingfacility structure and said second coupling facility structure inparallel.
 15. The method of claim 10, wherein at least one couplingfacility of the first coupling facility and the second coupling facilityincludes one or more structures that are not duplexed.
 16. A method ofcoupling a plurality of coupling facilities of a computing environment,said method comprising: selecting a first coupling facility and a secondcoupling facility to be coupled; providing at least one peer connectionbetween the first coupling facility and the second coupling facility tocouple the first coupling facility and the second coupling facility; andcoupling a third coupling facility to one of the first coupling facilityand the second coupling facility via at least one peer connection,wherein the third coupling facility need not be couple to the other ofthe first coupling facility and the second coupling facility.
 17. Themethod of claim 16, wherein the third coupling facility includes acoupling facility structure that is duplexed in the coupling facilitycoupled thereto, and wherein the coupling facility coupled theretoincludes a coupling facility structure that is duplexed in the othercoupling facility coupled thereto.
 18. A system of coupling a pluralityof coupling facilities of a computing environment, said systemcomprising: a first coupling facility and a second coupling facility tobe coupled, wherein at least one coupling facility of said firstcoupling facility and said second coupling facility comprises sharedstorage at includes active state information shareable by one or moredistributed applications of the computing environment; and at least onepeer connection between the first coupling facility and the secondcoupling facility to couple the first coupling facility and the secondcoupling facility.
 19. The system of claim 18, wherein a peer connectionof the at least one peer connection comprises an intersystem channel.20. The system of claim 18, wherein a peer connection of the at leastone peer connection comprises an integrated cluster bus.
 21. The systemof claim 18, wherein a peer connection of the at least one peerconnection comprises an internal coupling link.
 22. The system of claim18, wherein the first coupling facility and the second coupling facilityare coupled to provide duplexing of one or more coupling facilitystructures.
 23. The system of claim 18, wherein said at least one peerconnection comprises a plurality of peer connections, and whereinmultiple peer connections of the plurality of peer connections areconfigure as redundant connections between the first coupling facilityand the second coupling facility.
 24. The system of claim 22, wherein aninstance of one coupling facility structure of said one or more couplingfacility structures exists in said first coupling facility and saidsecond coupling facility, providing a first coupling facility structureinstance and a second coupling facility structure instance.
 25. Thesystem of claim 22, wherein at least one coupling facility of the firstcoupling facility and the second coupling facility includes one or morestructures that are not duplexed.
 26. The system of claim 24, furthercomprising means for maintaining synchronization of execution ofcommands issued against said first coupling facility structure instanceand said second coupling facility structure instance.
 27. The system ofclaim 26, wherein said means for maintaining synchronization comprisesmeans or exchanging one or more signals across one or more peerconnections of said at least one peer connection.
 28. The system ofclaim 26, wherein commands are issued against said first couplingfacility structure an said second coupling facility structure inparallel.
 29. The system of claim 23, further comprising: means forrecognizing a failure of a peer connection of the multiple peerconnections; and means for maintaining an exchange of information on oneor more surviving peer connections of the multiple peer connections. 30.A system of coupling a plurality of coupling facilities of a computingenvironment, said system comprising: a first coupling facility and asecond coupling facility to be coupled; at least one peer connectionbetween the first coupling facility and the second coupling facility tocouple the first coupling facility and the second coupling facility; andat least one peer connection to couple a third coupling facility to oneof the first coupling acuity and the second coupling facility, whereinthe third coupling facility need not be coupled to the other of thefirst coupling facility and the second coupling facility.
 31. The systemof claim 30, wherein the third coupling facility includes a couplingfacility structure that is duplexed in the coupling facility coupledthereto, and wherein the coupling facility coupled there includes acoupling facility structure that is duplexed in the other couplingfacility coupled thereto.
 32. A system of coupling a plurality ofcoupling facilities of a computing environment, said system comprising:means for selecting first coupling facility and a second couplingfacility to be coupled, wherein at least one coupling facility of saidfirst coupling facility and said second coupling facility comprisesshared storage that includes active state information shareable by oneor more distributed applications of the computing environment; and meansfor providing at least one peer connection between the first couplingfacility and the second coupling facility to couple the first couplingfacility and the second coupling facility.
 33. At least one programstorage device readable by a machine tangibly embodying at least oneprogram of instructions executable by the machine to perform a method ofcoupling a plurality of coupling facilities of a computing environment,said method comprising: selecting a first coupling facility and a secondcoupling facility to be coupled, wherein at least one coupling facilityof said first coupling facility and said second coupling facilitycomprises shared storage that includes active state informationshareable by one or more distributed applications of the computingenvironment; and providing at least one peer connection between thefirst coupling facility and the second coupling facility to couple thefirst coupling facility and the second coupling facility.
 34. The atleast one program storage device of claim 33, wherein a peer connectionof the at least one peer connection comprises an intersystem channel.35. The at least one program storage device of claim 33, wherein a peerconnection of the at least one peer connection comprises an integratedcluster bus.
 36. The at least one program storage device of claim 33,wherein a peer connection of the at least one peer connection comprisesan internal coupling link.
 37. The at least one program storage deviceof claim 33, wherein the first coupling facility and the second couplingfacility are coupled to provide duplexing of one or more couplingfacility structures.
 38. The at least one program storage device ofclaim 33, wherein said at least one peer connection comprise a pluralityof peer connections, and wherein multiple peer connections of theplurality of peer connections are configured as redundant connectionsbetween the first coupling facility and the second coupling facility.39. The at least one program storage device of claim 37, wherein aninstance of one coupling facility structure of said one or more couplingfacility structures exists in said first coupling facility and saidsecond coupling facility, providing a first coupling facility structureinstance and a second coupling facility structure instance.
 40. The atleast one program storage device of claim 38, wherein said methodfurther comprises: recognizing a failure of a peer connection of themultiple peer connections; and maintaining an exchange of information onone or more surviving peer connections of the multiple peer connections.41. The at least one program storage device of claim 39, wherein saidmethod further comprises maintaining synchronization of execution ofcommands issued against said first coupling facility structure instanceand said second coupling facility structure instance.
 42. The at leastone program storage device of claim 41, wherein said maintainingsynchronization comprises exchanging one or more signals across one ormore peer connections of said at least one peer connection.
 43. The atleast one program storage device of claim 41, wherein commands areissued against said first coupling facility structure and said secondcoupling facility structure in parallel.
 44. The at least one programstorage device of claim 37, wherein at least one coupling facility ofthe first coupling facility and the second coupling facility includesone or more structures that are not duplexed.
 45. At least one programstorage device readable by a machine tangibly embodying at least oneprogram of instructions executable by the machine to perform a method ofcoupling a plurality of coupling facilities of a computing environment,said method comprising: selecting a first coupling facility and a secondcoupling facility to be coupled; providing at least one peer connectionbetween the first coupling facility and the second coupling facility tocouple the first coupling facility and the second coupling facility; andcoupling a third coupling facility to one of the first coupling facilityand the second coupling facility via at least one peer connection,wherein the third coupling facility need not be coupled to the other ofthe first coupling facility and the second coupling facility.
 46. The atleast one program storage device of claim 45, wherein the third couplingfacility includes a coupling facility structure that is duplexed in thecoupling facility coupled thereto, and wherein the coupling facilitycoupled thereto includes a coupling facility structure that is duplexedin the other coupling facility coupled thereto.